[`Tokenizer`] Fix slow and fast serialization #26570
ArthurZucker
merged
114 commits into
huggingface:main
from
ArthurZucker:fix-main
fix
303a82cc
Merge branch 'main' of github.com:huggingface/transformers into fix-main
cbf179a0
last attempt
01e18dbc
current work
08a560a9
fix forward compatibility
23c95136
save all special tokens
0ae13ed4
Merge branch 'fix-main' of github.com:ArthurZucker/transformers into …
d887f681
current state
72ff80e4
revert additional changes
b7b7d13c
updates
36d5303b
remove tokenizer.model
ae938569
add a test and the fix
88ea3522
nit
ca98fbd0
revert one more break
3c22fbb5
fix typefield issue
dc93d5ed
quality
00997e99
more tests
61436349
fix fields for FC
907591f9
Merge branch 'fix-main' of github.com:ArthurZucker/transformers into …
5df5a834
Merge branch 'fix-main' of github.com:ArthurZucker/transformers into …
66ecb9ef
more nits?
0e7bd614
Merge branch 'fix-main' of github.com:ArthurZucker/transformers into …
381a0ecc
new additional changes
bf75334f
how
fafbbede
some updates
c6de7b2f
simplify all
9a6e750b
more nits
8c4ec2c1
revert some things to original
621ebae4
nice
6a6095e0
nits
e0e5deab
a small hack
92c77542
more nits
9fbbafe4
ahhaha
25e2df9b
Merge branch 'main' of github.com:huggingface/transformers into fix-main
2b18cc2a
fixup
078c94e4
ArthurZucker
changed the title
update
ef1e5985
make test run on ci
9bf12a89
use subtesting
e6d03813
update
112e4b16
ArthurZucker
commented
on 2023-10-11
Update .circleci/create_circleci_config.py
f794a91b
updates
65aa232d
Merge branch 'fix-main' of github.com:ArthurZucker/transformers into …
8ea095bb
fixup
efc5e7b5
nits
aa569b7d
replace typo
5ad55f31
fix the test
1c22269f
nits
3b936536
Merge branch 'main' of github.com:huggingface/transformers into fix-main
a2e977a4
update
1acf2dde
None max dif pls
2dde542e
a partial fix
9ebf76e5
had to revert one thing
6d2c00e1
test the fast
e4bcb5e9
updates
3d4bffdd
fixup
8bcb3451
and more nits
d9e5fad7
more fixes
fc34148c
update
83890944
Oupsy :eye:
78f1ac46
Merge branch 'main' of github.com:huggingface/transformers into fix-main
62eb8167
nits
5c1ae9c6
fix marian
df8ab6f6
on our way to heaven
677fcb28
LysandreJik
approved these changes
on 2023-10-12
Update src/transformers/models/t5/tokenization_t5.py
5a3407ef
fixup
856a43db
Update src/transformers/tokenization_utils_fast.py
a3cb498b
Update src/transformers/tokenization_utils_base.py
62cf2d07
fix phobert
fe8bba06
skip some things, test more
be68fc2a
nits
814d9786
fixup
f9697136
fix deberta
56b06190
update
f2a5447e
update
5d7bdab9
more updates
49dd8b2e
skip one test
3a03c77f
more updates
707a6888
fix camembert
bbfc382a
can't test this one
b6b8aedd
more good fixes
dac7b899
kind of a major update
b4ca44e7
fixup
52458257
more fixups
0724ebf3
fix pegasus and mpnet
066854a6
remove skipped tests
f646ab8a
fix phoneme tokenizer if self.verbose
53e23901
fix individual models
e0a967f8
update common tests
a3538710
update testing files
fbc4c4fe
all over again
64a6bc4f
nits
4219b32a
skip test for markup lm
48b937ab
fixups
d1a4537b
fix order of addition in fast by sorting the added tokens decoder
60173aad
proper defaults for deberta
84026022
correct default for fnet
d782bbd4
nits on add tokens, string initialized to special if special
05ab2c23
skip irrelevant herbert tests
bd6c5a5f
main fixes
8a267d3c
update test added_tokens_serialization
7bda15e9
the fix for bart like models and class instanciating
ac75cd3c
update bart
640885e8
nit!
45801c00
update idefix test
14c576f8
fix whisper!
2a78cf90
some fixup
6f28584e
fixups
c12656b9
revert some of the wrong chanegs
8f8c3f14
fixup
de51ef7f
fixup
0f0a3fe5
LysandreJik
approved these changes
on 2023-10-18
Merge branch 'main' of github.com:huggingface/transformers into fix-main
4b693b9a
skip marian
4b82043c
skip the correct tests
340df3dd
skip for tf and flax as well
f9fb43d0
Login to write a write a comment.
Login via GitHub