transformers
Update tokenizer mappings to use TokenizersBackend for additional models
#46091
Open
Overview
Commits
14
Changes
View On GitHub

Update tokenizer mappings to use TokenizersBackend for additional models #46091

itazap wants to merge 14 commits into main from tokenizers_backend_update
itazap
HuggingFaceDocBuilderDev
itazap itazap requested a review from ArthurZucker ArthurZucker 28 days ago
itazap itazap force pushed from 68f0144d to 3d7abeeb 28 days ago
itazap
itazap commented on 2026-05-20
itazap itazap force pushed from 05a70e5a to 2ba20bb3 19 days ago
itazap
github-actions
github-actions
ArthurZucker
ArthurZucker commented on 2026-06-02
itazap itazap requested a review from ArthurZucker ArthurZucker 13 days ago
itazap
itazap itazap force pushed from 765e75b2 to be52825f 5 days ago
itazap itazap force pushed from be52825f to d551fe04 2 days ago
itazap itazap force pushed from d551fe04 to 5e96632f 19 hours ago
itazap Update tokenizer mappings to use TokenizersBackend for additional models
7525b0ac
itazap aria
181d0fbe
itazap split scripts
64c7f092
itazap added test
5ed9b753
itazap Remove model-status-space
b6d2ccf2
itazap fix test
80299427
itazap fix test
76a9f9cf
itazap add MODEL_IDS_TO_TOKENIZERS_BACKEND list and test for models without …
a9bf262b
itazap new test
151bf03e
itazap reorg tests
9a9d87aa
itazap ruff
7af81d2e
itazap ruff
29fcf4bd
itazap match checkpoint path
45aee666
itazap fix test
76ff958f
itazap itazap force pushed from 5e96632f to 76ff958f 2 hours ago
github-actions
github-actions

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum