transformers
[`LlamaTokenizerFast`] Adds edge cases for the template processor
#26606
Merged
Overview
Commits
6
Changes
View On GitHub

[`LlamaTokenizerFast`] Adds edge cases for the template processor #26606

LysandreJik merged 6 commits into huggingface:main from ArthurZucker:fix-llama-eos
ArthurZucker
ArthurZucker make sure eos and bos are properly handled for fast tokenizer
a1472611
ArthurZucker fix code llama as well
e92c80c5
HuggingFaceDocBuilderDev
ArthurZucker nits
1162196e
LysandreJik LysandreJik requested a review from Rocketknight1 Rocketknight1 2 years ago
Rocketknight1
Rocketknight1 approved these changes on 2023-10-05
ArthurZucker fix the conversion script as well
e3d2206b
ArthurZucker fix failing test
04ddc0ac
ArthurZucker Merge branch 'main' of github.com:huggingface/transformers into fix-l…
f6faa873
LysandreJik
LysandreJik approved these changes on 2023-10-06
ArthurZucker
LysandreJik LysandreJik merged 9ad815e4 into main 2 years ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum