Fix MoE for V5 (#42456)
* remove zero_like + scatter
* fix mixtral moe
* fix other moe models as well
* fix ci
* fix modular mixtral
* fix qwen2_moe + qwen3_next
* fix device mismatch for qwen3_vl_moe to pass tests
* fix modular mixtral
* fix other models
* rm slow tokenizers (#40936)
* fixes missed
* gemma test fix
* refactor
* rm legacy from llama
* added renaming
* add _model
* update legacy
* update legacy
* fix docstring
* always load blank, then set _tokenizer if we have it
* new toks
* update all berttokenizer based models
* apply feedback - delete bert duplicates
* more models --> fast only
* more convert_slow models
* fix common test refs
* updating fast only tokenizers
* openai and pegasus
* enable sentencepiecebackend
* more models
* code gen
* t5
* code gen tests
* speecht5
* mbart
* mbart50
* more models
* more models
* layouglmv2
* update tests
* update tests
* update tests
* pretrainedtokenizer
* whisper
* whisper
* layoutxlm and storing backends
* refactor sentencepiecebackend and additional_special_tokens
* renaming tokenization_utils --> tokenization_python
* udpate tests
* bert test
* blenderbot
* clip
* codegen
* code_llama
* cohere
* deberata, deberat v2, funnel
* gpt2
* batch update tests
* pegasus qwen2 roberta
* more models
* layout tests
* some renaming
* fix references to utils_fast
* fix refs
* fix refs
* fix refs
* fix refs
* fix refs
* fix refs
* fix refs
* fix some tests
* regression
* fix refs
* fix refs
* missed the most crucial file in my last commit
* fix refs
* fix refs
* fix refs
* batch encode fix
* fix some tests
* BC for batch_decode bc too many refs
* more tests
* fix more tests
* fix for processors
* fixing more models
* deleted mbart50 by accident
* seamless m4t
* albert fix
* whisper
* layout3
* attempt to fix cached tokenizers on CI
* trying another fix on CI
* again try to work around CI
* bertweet
* tapas
* mbart50
* luke
* mluke
* markuplm
* markuplm
* fix some more auto tests
* some random model failures
* mistralcommontestser
* more fixes
* ref fix
* siglip
* marian
* plbart
* update utils toks
* seamless m4t
* roc bert
* udpate byt5 test
* xlm
* esm
* roformer
* code llama
* biogpt
* m2m100
* dpr and flaubert
* xlm and speech to text
* tok backend pass object
* tokenizer object pass
* wav2vec2
* wav2vec2
* cpmant
* update utils tokenizers
* cpmant
* bartpho
* test apply chat template assistant mask
* apply chat template video
* apply chat template assistant mask
* test torch
* update from slow in base and fix donut processor errors
* auto to point to tokenizers backend, fix kosmos2
* some non model fixes for old slow models that no longer have their own tokenizer file as they are the same as bert
* missed file from last commit
* idefics2
* fixup
* fixup
* pretrained tokenizer fast test update
* stash
* bad merged
* cherry pick more stuff that did not merge well
* fix gptsw3
* nit warn for now
* update error raising
* just ran fixup
* bring back bert legacy
* fix
* nit
* fix 56 errors on blenderbotsmall?
* 18 for blenderbotsmall
* tok auto
* missed clip
* fix tests
* something missed
* token healing
* tok common tests update - nonmodel
* try to fix non-model test in test_tokenization_utils
* fix hub tests
* try to fix hub tests
* custom vocab related fixed
* bert jap
* BERT JAP
* rename bert legacy to bert legacy
* Wav2vec2
* fix in tok python to update total vocab size - fixes speech t5
* blender bot small
* forgot test file
* test failures
* marian
* gpt2 tiktoken
* big bird / marian
* udop
* forgot couple changes
* test_serve fix
* missing import
* a couple processors fixes
* style partly
* fix to fetch tests ci
* Revert branch back to commit f5bc69ef state
* revert branch to styling
* update mistral after merge
* fixes for non model tests
* some processor test fixes
* more processor test fixes
* more processor fixes
* hub tests
* python tok utils
* fix hub test
* make style for now
* remove problemattic fic copies
* python utils/check_copies.py --fix_and_overwrite
* more styling
* fixup
* silence docstirng
* fix import?
* fix imports
* add the local test as well
* throw spm error
* llamas
* fix a couple tests
* broke ci
* broke ci
* broke ci
* broke ci
* add logs to debug gemma on ci
* gemma and llama
* gemma
* revert las commit
* gemma debug
* gemma debug
* gemma
* safely import spiece backend
* tok tests
* check none
* setup and qual
* ruff
* del dev files
* tok auto
* fill docstrings
* update auto
* blenderbot small nit
* add migration guide
* move mixtral patch to `TokenizersBackend`, move `TokenizerExtractor`
* rename MistralCommonTokenizer to MistralCommonB ackend
* nit
* fix failures
* fixup
* remoove one old test
* mark the slow one as slow
* very small fixes
* update auto mapping for missing ones
* fixup lorsd
* fixup doc and stuff
* should be the final fixe
* processing update
* update
* FIX or brute AI fix the llava test
* style
* slow?
* fix is offline mode?
* fix mt5
* One tok utils (#42462)
* consolidate python and utils tokenization files, they are copies
* ruff and ref
* Format
* fix cohere
* ?
* up
* am I dumbb?
* grumble
---------
Co-authored-by: Arthur <arthur.zucker@gmail.com>
* [loading/saving] Reverse all loading operations when saving (#42396)
* first shot
* default to reversing
* oupso
* oupsi 2
* oupsi 3
* fix renamed kwargs
* fix timm_wrapper
* remove fix_state_dict methods
* can do it all the time, with __init__ as well
* doc
* oupsi
* fix
* create helper
* fix annotation annoying isue
* small fix
* small fixes
* alright commit all that already
* oupsi
* the fix
* update quantizers
* this works
* the hardcoded regex got me hard....
* style
* the final one
* cleanup a bit
* better
* style
* oupsi readded it
* do it inside the ops instead - no need for full names anymore
* reverse quantizers and simplify signatures
* small thingy
* add no_grad decorator
* utils to rename keys
* oupssii again
* add test
* simplify nicely
* Fix T5 tests: use generation_config for generation parameters (#42419)
* pass the generation parameters to generate()
* fix use_task_specific_params to separate model.config and model.generation_config params
* fix style
* some fixes
* remove redundant check
* update expectation for llama_7b_bf16 on rocm
* Update tests/models/llama/test_modeling_llama.py
Co-authored-by: Rémi Ouazan <83456801+remi-or@users.noreply.github.com>
---------
Co-authored-by: Rémi Ouazan <83456801+remi-or@users.noreply.github.com>
* linting
* more fix to pass the CI tests
* fix lfm2 moe
* fix docstring
* fix docstring
* fix qwen like model
* fix flex olmo
* revert lfm2 moe config
* make fixup
* fix docstring
* fix conversion mapping
* fix inference of gpt-oss
* add some fixes to gpt-oss (but still not good)
* fix modular
* we need errors I think
* fix config issue
* this was fixed
---------
Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com>
Co-authored-by: Arthur <arthur.zucker@gmail.com>
Co-authored-by: Cyril Vallez <cyril.vallez@huggingface.co>
Co-authored-by: BADAOUI Abdennacer <106801897+Abdennacer-Badaoui@users.noreply.github.com>
Co-authored-by: Rémi Ouazan <83456801+remi-or@users.noreply.github.com>
