transformers
fix: improve processor loading performance by avoiding redundant tokenizer parsing
#44927

Merged

Commits

fix(tokenization_utils_tokenizers): avoid parsing full vocab in from_file when only post_processor/padding/truncation are needed

ydshieh committed 133 days ago
fix(tokenization_utils_tokenizers): fall back to from_file when model type is missing in tokenizer.json

ydshieh committed 133 days ago
fix(tokenization_utils_tokenizers): restrict minimal tokenizer optimization to BPE/WordPiece/WordLevel only

ydshieh committed 133 days ago
fix(tokenization_utils_tokenizers): add comment explaining why Unigram and older formats fall back to from_file

ydshieh committed 133 days ago
apply suggestions

ydshieh committed 133 days ago
fix

ydshieh committed 133 days ago