transformers
0566b6f5 - Patch MistralCommonTokenizer (#41439)

Commit
121 days ago
Patch MistralCommonTokenizer (#41439) * Fix token_to_id and add add_generation_prompt * Fix spm download * Refactor spm * Try another possibly non-gated spm * Improve get_vocab * lint * Improve get_vocab * Add warn to piece_to_id * Improve from_pretrained raise and revert model spm * Revert fast
Author
Parents
Loading