transformers
Fix convert_tokens_to_ids performance regression for slow tokenizers (#46315)
#46323

Merged

Fix convert_tokens_to_ids performance regression for slow tokenizers (#46315) #46323

itazap merged 2 commits into huggingface:main from ishan-1010:fix/added-tokens-encoder-perf-46315

Fix convert_tokens_to_ids performance regression for slow tokenizers …

1947a3df

itazap commented on 2026-06-02

Remove regression test per reviewer feedback

d2eaaf78

ishan-1010 requested a review from

itazap 91 days ago

itazap approved these changes on 2026-06-03

itazap merged d3f05911 into main 90 days ago

Reviewers

itazap

Assignees

No one assigned

Labels

None yet

Milestone

No milestone