transformers
Load a pretrainedfast tokenizer if fast=true and tokenizer.json exists
#33751
Open
Open
Load a pretrainedfast tokenizer if fast=true and tokenizer.json exists #33751
itazap
wants to merge
6 commits into
main
from
load_pretrainedfast_auto
add support for loading a pretrainedfast tokenizer if fast=true and t…
d9378690
itazap
changed the title
diff idea
7f2e583f
rm siglip change
3db349e6
ArthurZucker
commented
on 2024-09-27
vignesh1507
approved these changes
on 2024-09-27
parse mapping instead
f4d7160d
check if can index
88b140e8
vignesh1507
approved these changes
on 2024-09-29
ArthurZucker
commented
on 2024-10-03
ArthurZucker
commented
on 2024-10-04
check if repo has tokenizer.json file, then load PreTrainedTokenizerFast
b6ae24fc
Login to write a write a comment.
Login via GitHub