PR #26576 Update tokenization_code_llama_fast.py

Update tokenization_code_llama_fast.py #26576

ArthurZucker merged 3 commits into huggingface:main from andyl98:codellama_fast_tokenizer_infilling_fix

Update tokenization_code_llama_fast.py

963200be

ArthurZucker commented on 2023-10-04

Update test_tokenization_code_llama.py

1596d9aa

Update test_tokenization_code_llama.py

3048f3ee

andyl98 requested a review from

ArthurZucker 2 years ago

ArthurZucker approved these changes on 2023-10-06

ArthurZucker merged 65aabafe into main 2 years ago

Reviewers

ArthurZucker

Assignees

No one assigned

Labels

None yet

Milestone

No milestone