PR #25793 [`LlamaTokenizer`] `tokenize` nits.

[`LlamaTokenizer`] `tokenize` nits. #25793

ArthurZucker merged 5 commits into huggingface:main from ArthurZucker:llama-tokenizer-nits

return when length is zero

2b00b0b6

Add tests

8f23e85c

Co-authored-by: avnishn

d7732c60

ArthurZucker force pushed to d7732c60 2 years ago

codeLlama doc should not be on Main

4ec79c3a

update test

2e943193

ArthurZucker marked this pull request as ready for review 2 years ago

ArthurZucker requested a review from

LysandreJik 2 years ago

ArthurZucker requested a review from

amyeroberts 2 years ago

amyeroberts approved these changes on 2023-08-29

LysandreJik approved these changes on 2023-08-29

ArthurZucker merged 5b5ee235 into main 2 years ago

Reviewers

LysandreJik

amyeroberts

Assignees

No one assigned

Labels

None yet

Milestone

No milestone