transformers
efe40496 - Update tokenizer mappings to use TokenizersBackend for additional models (#46091)

Commit
29 days ago
Update tokenizer mappings to use TokenizersBackend for additional models (#46091) * Update tokenizer mappings to use TokenizersBackend for additional models * aria * split scripts * added test * Remove model-status-space * fix test * fix test * add MODEL_IDS_TO_TOKENIZERS_BACKEND list and test for models without tokenizer_class Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * new test * reorg tests * ruff * ruff * match checkpoint path * fix test * workaround for CI read-only file system error --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Author
Parents
Loading