Refactor-tokenization-more #42563
ArthurZucker
merged
124 commits into
main
from
refactor-tokenization-more
On commit to bind them all
7fd3d4f0
Merge branch 'main' of github.com:huggingface/transformers into refac…
8fe488e7
nits
84e7717c
smnall update
5743c9c2
elif
8edc595f
super small nit
f8d0a657
BPE!
8cc4f719
fix
90cfd47a
up up up
6ef7df1d
fix?
d2ead7ee
one typo
d8e7b6e1
per model updates
974f0bd0
more model specific updates
fd4673e7
more per model updates
1521dace
more model specific updates
1ab527b4
simplify default merges
a95b84f3
fiuxp
cdd7294e
update
b6ac1883
update
19cf8338
style
10cacec4
fix
36dfa350
fix colpali
92158f0b
nits
5eb502e7
simpler regex + big shitty bird
c399c3a6
Merge branch 'main' of github.com:huggingface/transformers into refac…
c7693003
fixup and fix
8faa347d
fix codellama
bcadd99f
up
89c0147b
fix pop on none
3a8a2c24
fix parkeet
89430214
fix llama
9fc484ef
big fixup
b099093f
fix markul lm
0b346278
update common
5dc21242
fix mbart
a2ed23f8
fix seamlessm4T
c94f1c3d
fix comment
bb97d54f
torch tests
6796ec0a
nnits and revert UNK idx change
df4dbdfa
oh only one deberta
4d6269a5
torch tests
d7968403
add convert from spm per model!
de3ec0ac
Merge branch 'refactor-tokenization-more' of github.com:huggingface/t…
32cb8a24
fix last 2 for pegasus
99ff6cd0
fix torch tests
c292f1b6
fixes
8621b194
Merge branch 'refactor-tokenization-more' of github.com:huggingface/t…
f74474f4
fix tests
0cf15c8f
check versioned files
0764ffaa
fix processor auto test
32ef79e5
fix custom tok clip
5121b416
try this fix
1355f09a
modeling rag
5fd30335
fix rag
ffba8744
roformer the Tokenizers way
255ea31b
up
b3b8bc41
updatge
422021e0
Merge branch 'main' of github.com:huggingface/transformers into refac…
81589daf
Merge branch 'refactor-tokenization-more' of github.com:huggingface/t…
41e5590a
fix unk
00ca3232
update
9a9189ce
fix roberta
b0939308
if there is no mapped class and no tokenizer.json its fucked -> just …
a665ad5d
fix the rest
1e9d7325
fix copies
60da2e85
fix doc and copies
9e5ea38b
fix mbart50
feeda9f9
fix deberta_v2 test
54325444
fix and simplify whisper :)
23d2cb9b
fix big bird default was worng
107b7d75
fix final
4e9a4b0b
fixup
eabf3c41
small nit
0ceba45f
a weird way to fix fuyu?
c7877068
default xlm roberta to fix kosmo behaviour!
c013b393
remove small errors
05ae154d
last fix?
9fa4ecf7
fix pixtral
e5f1cc33
style
4ec42008
fix
838c6c23
quality ta radce
0da379ee
Merge branch 'main' of github.com:huggingface/transformers into refac…
98a5d30a
fix?
ddaf841d
remove something
c48dfde3
remov one code that shouldd not have been there!
7d2afb27
fix ?
c7520dad
fixup
40395a63
update
d804136a
fix for custom code
69ffef0f
add a custom model path to make sure custom stuff is registe
af406a8c
fix trust remote code
0a401995
exceeded
9c20d628
don't
7a12795c
ouppsy for cohere
7282e511
why is this one also affected
4e14adf6
fixup
1e3b19b3
Merge branch 'main' of github.com:huggingface/transformers into refac…
0d006fbe
fixup
0557cbb7
nits
08b67f60
fix idefics3 tests
9d991dc9
okay read the processor
76fc45f0
fix the layout.... models
62387f7b
nits
74a87df3
codellama needs the bos passed
c1859453
fix dpr
eeffd1a3
fix?
e065d248
fixup
2be41c33
distilbert defaults
e119fda7
fix
2363ea46
clvp update to PythonTokenizer
ae920b96
bloom
1dbfc94f
style
4cee9faa
Merge branch 'refactor-tokenization-more' of github.com:huggingface/t…
6436b39b
layoutxlm
fdd54d42
Merge branch 'refactor-tokenization-more' of github.com:huggingface/t…
804c382d
style
161aa266
olmo
9e39d3f1
only pop when we don't convert from tokenizer.json
22ee211e
Merge branch 'refactor-tokenization-more' of github.com:huggingface/t…
9fd05521
fixup
2096633f
hub issue
47a29174
id
696b81fe
fix
0ec5befc
Merge branch 'main' into refactor-tokenization-more
328bfc1d
tomaarsen
commented
on 2026-01-26
Login to write a write a comment.
Login via GitHub