llama : fix pre-tokenization of non-special added tokens #8228
compilade
merged
14 commits into
master
from
compilade/fix-mpt-pretok
llama : fix mpt and olmo pre-tokenizer
db2ffd51
jaime-m-p
commented
on 2024-07-01
Merge branch 'master' into compilade/fix-mpt-pretok
ac0f33c9
llama : pre-tokenize non-special user-defined tokens first
d5d30b20
Merge branch 'master' into compilade/fix-mpt-pretok
6b961e3d
llama : fix detection of control-like user-defined tokens
56df1fcd
convert_hf : identify which user-defined tokens are control tokens
6e351e04
convert_hf : identify more added control tokens for SPM tokenziers
f9d42c59
compilade
changed the title
bartowski1182
commented
on 2024-07-08
llama : fix Viking pre-tokenizer regex
31a1b0ee
llama : fix command-r detokenization
d6fe269c
jaime-m-p
commented
on 2024-07-08
jaime-m-p
commented
on 2024-07-09
convert_hf : reduce usages of the UNKNOWN token type
d4df7858
llama : add UNKNOWN tokens in the special tokens cache
98edea60
Merge branch 'master' into compilade/fix-mpt-pretok
afa61198
convert_hf : reduce usages of UNKNOWN for InternLM2
1caa20fc
ggerganov
approved these changes
on 2024-07-12
test-tokenizer-random : reduce potential confilcts with #8379
59ce8531
Reviewers
Assignees
No one assigned
Labels
generation quality
testing
bugfix
python
Review Complexity : Medium
merge ready
Milestone
No milestone
Login to write a write a comment.
Login via GitHub