transformers
b6ae24fc - check if repo has tokenizer.json file, then load PreTrainedTokenizerFast

Commit

1 year ago

check if repo has tokenizer.json file, then load PreTrainedTokenizerFast

References

load_pretrainedfast_auto

#33751 - Load a pretrainedfast tokenizer if fast=true and tokenizer.json exists

Author

Ita Zaporozhets

Parents

Loading