[`NllbTokenizer`] refactor with added tokens decoder #27717
ArthurZucker
merged
36 commits into
main
from
refactor-nllb
refactor with addedtokens decoder
76dd3251
style
862d6d1a
get rid of lang code to id
0ae401f7
style
b230b6bb
keep some things for BC
329569f7
update tests
ee08b8d8
avidale
commented
on 2023-11-27
add the mask token at the end of the vocab
d5b31958
nits
086000a5
nits
23aa8409
Merge branch 'main' of github.com:huggingface/transformers into refac…
dbd80f68
Merge branch 'main' of github.com:huggingface/transformers into refac…
5ec00d27
fix final tests
8a14c0ba
Merge branch 'main' of github.com:huggingface/transformers into refac…
a77609f3
style
f79abb76
nits
f74b89cc
amyeroberts
commented
on 2024-01-17
amyeroberts
approved these changes
on 2024-01-17
Update src/transformers/models/nllb/tokenization_nllb_fast.py
337f0687
nits
c79212ec
style?
00943948
ArthurZucker
commented
on 2024-01-18
Update src/transformers/convert_slow_tokenizer.py
ea9a78ef
make it a tad bit more custom
424c8e4e
ruff please stop
eac39479
Update
6bf26310
Update
fa8ab617
oupts
db35a5d4
ouft
0d19d4c3
nites
3e01ea7a
test
ea6906c5
Merge branch 'main' of github.com:huggingface/transformers into refac…
dc56b4ab
fix the remaining failing tests
4f21f5e4
style
0f44c396
fix failing test
a0383694
ficx other test
7172ad76
Merge branch 'main' of github.com:huggingface/transformers into refac…
93c02de8
avidale
commented
on 2024-02-12
temp dir + test the raw init
f2ba78f0
update test
5b08ee00
style
ed86fa6d
Login to write a write a comment.
Login via GitHub