transformers
9ad815e4 - [`LlamaTokenizerFast`] Adds edge cases for the template processor (#26606)

Commit
2 years ago
[`LlamaTokenizerFast`] Adds edge cases for the template processor (#26606) * make sure eos and bos are properly handled for fast tokenizer * fix code llama as well * nits * fix the conversion script as well * fix failing test
Author
Parents
Loading