transformers
[ `TokenizationUtils`] Fix `add_special_tokens` when the token is already there
#28520

Merged

[ `TokenizationUtils`] Fix `add_special_tokens` when the token is already there #28520

ArthurZucker merged 6 commits into main from tokenizer-nits

fix adding special tokens when the token is already there.

0313833d

add a test

d1d22125

add a test

b39c4e0f

ArthurZucker changed the title ~~Tokenizer-nits~~ [ `TokenizationUtils`] Fix `add_special_tokens` when the token is already there 2 years ago

nit

624abe3f

ArthurZucker requested a review from

amyeroberts 2 years ago

ArthurZucker marked this pull request as ready for review 2 years ago

fix the test: make sure the order is preserved

64fa9f58

amyeroberts approved these changes on 2024-01-16

Update tests/test_tokenization_common.py

1949bdcf

ArthurZucker merged 716df5fb into main 2 years ago

ArthurZucker deleted the tokenizer-nits branch 2 years ago

Reviewers

amyeroberts

Assignees

No one assigned

Labels

None yet

Milestone

No milestone