transformers
9ad815e4
- [`LlamaTokenizerFast`] Adds edge cases for the template processor (#26606)
Go
Login via GitHub
Home
Pricing
FAQ
Install
Login
via GitHub
Commit
View On
GitHub
Commit
2 years ago
[`LlamaTokenizerFast`] Adds edge cases for the template processor (#26606) * make sure eos and bos are properly handled for fast tokenizer * fix code llama as well * nits * fix the conversion script as well * fix failing test
References
#26606 - [`LlamaTokenizerFast`] Adds edge cases for the template processor
Author
ArthurZucker
Parents
27597fea
Loading