transformers
e34da3ee - [`LlamaTokenizerFast`] Refactor default llama (#28881)

Commit
1 year ago
[`LlamaTokenizerFast`] Refactor default llama (#28881) * push legacy to fast as well * super strange * Update src/transformers/convert_slow_tokenizer.py * make sure we are BC * fix Llama test * nit * revert * more test * style * update * small update w.r.t tokenizers * nit * don't split * lol * add a test for `add_prefix_space=False` * fix gemma tokenizer as well * update * fix gemma * nicer failures * fixup * update * fix the example for legacy = False * use `huggyllama/llama-7b` for the PR doctest * nit * use from_slow * fix llama
Author
Parents
Loading