transformers
Fix models for which we don't have a dedicated tokenizer class, and the listed one is incorrect
#45936
Open
Open
Fix models for which we don't have a dedicated tokenizer class, and the listed one is incorrect #45936
itazap
wants to merge
25 commits into
main
from
check_repo_model_tokenizers
we don't check the tokenization of all the model paths we have in the…
8cf2fe34
fixessss
7723158b
rerun with fixes
b90c6fec
deepseek-r1 fix and other cleanup
bf2a837f
sort
31b1394c
fixessss and style
42c6ed64
clean up scripts
c5dfd1f5
clean scripts
cce2efc7
org output
41385ed6
rm unrelated file
e8a3ad2d
clearer script names
2703d891
clean up output
ba05ad46
update output
9ec670b6
albert
f0ad6780
final rerun
500444b0
Merge branch 'main' into check_repo_model_tokenizers
0501a670
adding original report pre fix
702336de
updated report
ea781602
update test
e4b61804
revert changes to albert
5671fc49
update test
d4b6028e
albert
83476364
Merge remote-tracking branch 'origin/main' into check_repo_model_toke…
0281eaa8
update debug script
5911ce96
albert precompiled charsmap alternative
ec9c26df
itazap
commented
on 2026-06-02
Login to write a write a comment.
Login via GitHub