llama.cpp
Detokenizer fixes
#8039
Merged
Go
Login via GitHub
Home
FAQ
Install
Login
via GitHub
Overview
Commits
30
Changes
View On
GitHub
Detokenizer fixes
#8039
jaime-m-p
merged 30 commits into
ggml-org:master
from
jaime-m-p:detokenizer
Add llama_detokenize()
eea8dfab
Using llama_tokenize() in tests
d779bab4
jaime-m-p
added
bugfix
jaime-m-p
marked this pull request as draft
2 years ago
Using llama_tokenize() in tests
40a66606
Fix tokenizer tests
16a7503d
minor: confusing hexadecimal codepoint
03dbcc89
Clean old known problematic codepoints
071bf42f
github-actions
added
testing
Update bruteforce random tests
064b35ea
github-actions
added
python
Fix add_space_prefix, set false by default
503b7531
Remove previous space
0cc6593f
Remove previous space
6d233bc1
github-actions
added
examples
Add tokenizer flag: clean_up_tokenization_spaces
b452e826
mofosyne
added
Review Complexity : Medium
tests: unexpected vocab type as test fail instead of error
9af762c0
tests: gracefully exit threads
0cf2989b
tets: skip unicode surrogaes and undefined
38d54b3c
Fix detokenizer():
44c86484
Do not remove space when decoding special tokens
9eb0fca0
style: remove trailing whitespace
12e2c317
Bugfix: custom regexs splits undefined unicode codepoints
95a0df55
Update brute force test:
4a28063b
Symetric params for llama_tokenize() and llama_detokenize()
9854a9cd
Better leading space removal
107923cd
Update bruteforce test
68220fea
style : remove spaces
98fc1823
Merge commit 'f8c4c073' into detokenizer
8072089c
'viking' detokenizer clean spaces
8f5e1e0c
Better leading space removal
2f150197
Update bruteforce test: header files location
11ac641c
Update bruteforce test: add more models
4db8c0d5
style: spaces
906476f6
jaime-m-p
marked this pull request as ready for review
2 years ago
jaime-m-p
requested a review
from
ggerganov
2 years ago
ggerganov
approved these changes on 2024-07-05
style: spaces
0137683e
jaime-m-p
merged
213701b5
into master
2 years ago
Login to write a write a comment.
Login via GitHub
Reviewers
ggerganov
Assignees
No one assigned
Labels
testing
examples
bugfix
python
Review Complexity : Medium
Milestone
No milestone
Login to write a write a comment.
Login via GitHub