transformers
Fix bug in slow tokenizer conversion, make it a lot faster
#24266

Merged

Fix bug in slow tokenizer conversion, make it a lot faster #24266

amyeroberts merged 3 commits into huggingface:main from main

Make conversion faster, fix None vs 0 bug

0585c5a8

Add second sort for consistency

9ee18921

amyeroberts approved these changes on 2023-06-14

ArthurZucker commented on 2023-06-14

Update src/transformers/convert_slow_tokenizer.py

33bc7226

ArthurZucker approved these changes on 2023-06-14

amyeroberts merged 6793f0cf into main 2 years ago

Reviewers

ArthurZucker

amyeroberts

Assignees

No one assigned

Labels

None yet

Milestone

No milestone