transformers
b6055550 - [`Mistral Tokenizers`] Fix tokenizer detection (#42389)

Commit
21 days ago
[`Mistral Tokenizers`] Fix tokenizer detection (#42389) * fix * sanity check * style * comments * make it v5 explicit * make explicit fixes possible in local tokenizers * remove hub usage on local * fix * extend test for no config case * move mistral patch outside to separate fn * fix local path only * add a tes * make sure test does not pass before this PR * styling * make sure it exists * fix * fix * rename * up * last nit i hope lord --------- Co-authored-by: Arthur <arthur.zucker@gmail.com>
Author
Parents
Loading