Placeholder tokens update #44467
itazap
wants to merge
48 commits into
main
from
placeholder_tokens_update
update deepseek v2 for tokenizers v5
7db52901
adding remote code fix
5ef00618
fix deepseek name
7be0c578
handle spm conversion from proto only when overriding bad_models
1d227013
add script to compare xlni and code_search_net output of 2 tokenizers
43a07e1d
tiktoken models support
ffb5f095
fix tests
c1a3a0dd
testssss
43078316
fix gemma
1bbe2578
apply some feedback
b5d0badc
paligemma processor tests fix
b7547e42
add relevant changes from #44298
f5fd8400
json serializable fix
2b9efdf0
add more xlni cases
e1411666
t5 fix
ae253810
ruff check code quality
77120a21
missed file for t5 test fix
3b053b05
modular failures
a5542cca
other modular fixes
95bba6ca
tiktoken.model test
7d46f77f
more feedback updates!
be29c606
fixing models so AutoTokenizer == TokenizersBackend - aligning with c…
53753c31
seamless m4t
47457459
missed the most important files
cbda0cac
Revert "missed the most important files"
e5c8a2f7
undo changes to big bird , bert, seamless
8bf6df09
setup and qual
df12cc4e
lasr
08b91c68
t5
a7c2435b
dpr bert
d5e9aba1
xlmroberta
ceeb3197
reformer
b512fc70
nllb
0c958427
style and shit
34d83ed7
update
5c8af86f
fix
4d068711
extract the charsmap
2159e92f
fix mbart?
db0c5b58
style
31ff32d2
nllb and test tok common read spm precompiled charsmap
083ec509
fix whisper?
a4fc098f
Merge branch 'bad_models_update' of github.com:huggingface/transforme…
2710fadf
nllb
969c0fc6
checked on v4!
47a772ae
Merge branch 'bad_models_update' of github.com:huggingface/transforme…
8039c2b6
fix repo
e3d30250
Replace placeholder tokens as specified in added_tokens_decoder
64b0f05b
Revert "Replace placeholder tokens as specified in added_tokens_decoder"
8a293c84
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub