transformers

fd6bc380 - [vllm + v5 fix] handle TokenizersBackend fallback properly for v5 (#44255)

[vllm + v5 fix] handle TokenizersBackend fallback properly for v5 (#44255) * update deepseek v2 for tokenizers v5 * adding remote code fix * fix deepseek name * handle spm conversion from proto only when overriding bad_models * add script to compare xlni and code_search_net output of 2 tokenizers * tiktoken models support * fix tests * testssss * fix gemma * apply some feedback * paligemma processor tests fix * add relevant changes from #44298 * json serializable fix * add more xlni cases * t5 fix * ruff check code quality * missed file for t5 test fix * modular failures * other modular fixes * tiktoken.model test * more feedback updates! * fixing models so AutoTokenizer == TokenizersBackend - aligning with converters * seamless m4t * missed the most important files * Revert "missed the most important files" This reverts commit 8fcc625b6843a11bc9794a847b445948cd310f4f. * undo changes to big bird , bert, seamless * setup and qual * lasr * t5 * dpr bert * xlmroberta * reformer * nllb * style and shit * update * fix * extract the charsmap * fix mbart? * style * nllb and test tok common read spm precompiled charsmap * fix whisper? * nllb * checked on v4! * fix repo * fix lasr * style --------- Co-authored-by: ita.zaporozhets@huggingface.co <ita_zaporozhets@ip-26-0-164-75.ec2.internal> Co-authored-by: ita.zaporozhets@huggingface.co <ita_zaporozhets@ip-26-0-170-31.ec2.internal> Co-authored-by: Arthur <arthur.zucker@gmail.com>