transformers
fix: improve processor loading performance by avoiding redundant tokenizer parsing
#44927

Merged

fix: improve processor loading performance by avoiding redundant tokenizer parsing #44927

ydshieh merged 6 commits into main from faster_processor_loading

ydshieh commented on 2026-03-23

ydshieh requested a review from

ArthurZucker 132 days ago

ydshieh requested a review from

itazap 132 days ago

tarekziade commented on 2026-03-23

ArthurZucker approved these changes on 2026-03-23

ydshieh force pushed from 099e8755 to 05b3467a 132 days ago

fix(tokenization_utils_tokenizers): avoid parsing full vocab in from_…

0b8633a7

fix(tokenization_utils_tokenizers): fall back to from_file when model…

4224c0a9

fix(tokenization_utils_tokenizers): restrict minimal tokenizer optimi…

280d1040

fix(tokenization_utils_tokenizers): add comment explaining why Unigra…

29b315ef

apply suggestions

9bea0f9c

ydshieh force pushed from 05b3467a to 9bea0f9c 132 days ago

ydshieh commented on 2026-03-23

ydshieh enabled auto-merge 132 days ago

fix

87a40fac

ydshieh merged b1527a32 into main 132 days ago

ydshieh deleted the faster_processor_loading branch 132 days ago

Reviewers

ArthurZucker

tarekziade

itazap

Assignees

No one assigned

Labels

None yet

Milestone

No milestone