llama.cpp
Detokenizer fixes
#8039
Merged

Detokenizer fixes #8039

jaime-m-p merged 30 commits into ggml-org:master from jaime-m-p:detokenizer
jaime-m-p
Add llama_detokenize()
eea8dfab
Using llama_tokenize() in tests
d779bab4
jaime-m-p jaime-m-p added bugfix
jaime-m-p jaime-m-p marked this pull request as draft 2 years ago
Using llama_tokenize() in tests
40a66606
jaime-m-p
Fix tokenizer tests
16a7503d
minor: confusing hexadecimal codepoint
03dbcc89
Clean old known problematic codepoints
071bf42f
github-actions github-actions added testing
jaime-m-p
Update bruteforce random tests
064b35ea
github-actions github-actions added python
jaime-m-p
Fix add_space_prefix, set false by default
503b7531
Remove previous space
0cc6593f
Remove previous space
6d233bc1
github-actions github-actions added examples
jaime-m-p
Add tokenizer flag: clean_up_tokenization_spaces
b452e826
mofosyne mofosyne added Review Complexity : Medium
cmp-nct
tests: unexpected vocab type as test fail instead of error
9af762c0
tests: gracefully exit threads
0cf2989b
tets: skip unicode surrogaes and undefined
38d54b3c
Fix detokenizer():
44c86484
jaime-m-p
jaime-m-p
cmp-nct
Harsha-Nori
Do not remove space when decoding special tokens
9eb0fca0
style: remove trailing whitespace
12e2c317
Bugfix: custom regexs splits undefined unicode codepoints
95a0df55
Update brute force test:
4a28063b
jaime-m-p
Symetric params for llama_tokenize() and llama_detokenize()
9854a9cd
Better leading space removal
107923cd
Update bruteforce test
68220fea
jaime-m-p
Harsha-Nori
style : remove spaces
98fc1823
Merge commit 'f8c4c073' into detokenizer
8072089c
'viking' detokenizer clean spaces
8f5e1e0c
Better leading space removal
2f150197
Update bruteforce test: header files location
11ac641c
Update bruteforce test: add more models
4db8c0d5
jaime-m-p
style: spaces
906476f6
jaime-m-p jaime-m-p marked this pull request as ready for review 2 years ago
jaime-m-p
jaime-m-p jaime-m-p requested a review from ggerganov ggerganov 2 years ago
ggerganov
ggerganov approved these changes on 2024-07-05
jaime-m-p style: spaces
0137683e
jaime-m-p jaime-m-p merged 213701b5 into master 2 years ago
jaime-m-p

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
Milestone