transformers

d3f05911 - Fix convert_tokens_to_ids performance regression for slow tokenizers (#46315) (#46323)

Fix convert_tokens_to_ids performance regression for slow tokenizers (#46315) (#46323) * Fix convert_tokens_to_ids performance regression for slow tokenizers (#46315) Slow (PreTrainedTokenizer) tokenizers resolved added-vocabulary tokens through the added_tokens_encoder property, which rebuilds and re-sorts the whole added-token mapping on every access and is read twice per token. That made convert_tokens_to_ids roughly O(T * N * log N) for N added tokens, a regression from the v5 tokenizer refactor (#40936). Read the maintained _added_tokens_encoder cache instead, restoring the v4 behavior that every other method in the file already relies on. Adds a network-free regression test using CTRLTokenizer. * Remove regression test per reviewer feedback