[`Core Tokenization`] Support a fix for spm fast models #26678
ArthurZucker
merged
62 commits into
huggingface:main
from
ArthurZucker:fix-spm-fast-models
fix
303a82cc
Merge branch 'main' of github.com:huggingface/transformers into fix-main
cbf179a0
last attempt
01e18dbc
current work
08a560a9
fix forward compatibility
23c95136
save all special tokens
0ae13ed4
Merge branch 'fix-main' of github.com:ArthurZucker/transformers into …
d887f681
current state
72ff80e4
revert additional changes
b7b7d13c
updates
36d5303b
remove tokenizer.model
ae938569
add a test and the fix
88ea3522
nit
ca98fbd0
revert one more break
3c22fbb5
fix typefield issue
dc93d5ed
quality
00997e99
more tests
61436349
fix fields for FC
907591f9
Merge branch 'fix-main' of github.com:ArthurZucker/transformers into …
5df5a834
Merge branch 'fix-main' of github.com:ArthurZucker/transformers into …
66ecb9ef
more nits?
0e7bd614
Merge branch 'fix-main' of github.com:ArthurZucker/transformers into …
381a0ecc
new additional changes
bf75334f
how
fafbbede
some updates
c6de7b2f
the fix
104b03a0
where do we stand
c2b03c1d
Merge branch 'main' of github.com:huggingface/transformers into fix-s…
a088e205
ArthurZucker
commented
on 2023-11-13
Merge branch 'main' of github.com:huggingface/transformers into fix-s…
12d8d9ac
nits
332ecb5e
nits
2fb45139
revert unrelated changes
1d3f38ca
nits nits nits
5045c88f
styling
f173225c
don't break llama just yet
2947e2b2
revert llama changes
f62694c6
safe arg check
2ec474bc
fixup
10628975
ArthurZucker
changed the title
Add a test for T5
ace0d0fa
Necessary changes
1f621b21
Tests passing, added tokens need to not be normalized. If the added t…
20ca1bee
Add even more tests, when normalization is set to True (which does no…
080186c2
Merge branch 'main' of github.com:huggingface/transformers into fix-s…
6388dd81
Add even more tests, when normalization is set to True (which does no…
6884ee44
Update to main
31d34bde
Merge branch 'main' of github.com:huggingface/transformers into fix-s…
ca2bfb5e
nits
79e51d26
ArthurZucker
commented
on 2023-12-13
fmt
efdd11d4
more and more test
575bd5a9
comments
cfd3e8d1
ArthurZucker
commented
on 2023-12-18
Merge branch 'main' of github.com:huggingface/transformers into fix-s…
06260b1d
revert change as tests are failing
8f309e5d
make the test more readble
084c279a
nits
7fdee17e
refactor the test
6f476ccc
nit
1c3f4778
updates
305e52cc
simplify
714dd611
style
18e0d4a3
style
fa87e6cf
style convert slow
49ac0cb1
LysandreJik
approved these changes
on 2024-01-18
ArthurZucker
commented
on 2024-01-18
Update src/transformers/convert_slow_tokenizer.py
a709039b
Login to write a write a comment.
Login via GitHub