transformers
Fix somehow incorrect model - tokenizer mapping in tokenization testing
#19872

Merged

Fix somehow incorrect model - tokenizer mapping in tokenization testing #19872

ydshieh merged 4 commits into main from fix_tok

Fix model-tokenizer mapping

17eaf404

fix

4d15aae2

fix

32ccba26

fix

b6962f63

ydshieh changed the title ~~Fix the not very correct model - tokenizer mapping in the testing~~ Fix somehow incorrect model - tokenizer mapping in tokenization testing 3 years ago

ydshieh requested a review from

sgugger 3 years ago

ydshieh commented on 2022-10-25

sgugger approved these changes on 2022-10-25

ydshieh merged f9257843 into main 3 years ago

ydshieh deleted the fix_tok branch 3 years ago

Reviewers

sgugger

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

transformers Fix somehow incorrect model - tokenizer mapping in tokenization testing #19872 Merged

Fix somehow incorrect model - tokenizer mapping in tokenization testing #19872

transformers
Fix somehow incorrect model - tokenizer mapping in tokenization testing
#19872

Merged