transformers
5b5ee235 - [`LlamaTokenizer`] `tokenize` nits. (#25793)

Commit
2 years ago
[`LlamaTokenizer`] `tokenize` nits. (#25793) * return when length is zero * Add tests Co-authored-by: Avnish Narayan <38871737avnishn@users.noreply.github.com> * Co-authored-by: avnishn <38871737+avnishn@users.noreply.github.com> * codeLlama doc should not be on Main * update test --------- Co-authored-by: Avnish Narayan <38871737avnishn@users.noreply.github.com>
Author
Parents
Loading