transformers
[`CodeLlamaTokenizer`] Nit, update __init__ to make sure the AddedTokens are not normalized because they are special
#27359
Merged
Merged
[`CodeLlamaTokenizer`] Nit, update __init__ to make sure the AddedTokens are not normalized because they are special #27359
ArthurZucker
merged
4 commits into
main
from
nit-codellama
make sure tokens are properly initialized for codellama slow
d7f572c1
add m ore pretrained models
5b147d38
style
4d7ae8c8
LysandreJik
approved these changes
on 2023-11-09
test more tokenizers checkpoints
fea6f59f
LysandreJik
approved these changes
on 2023-11-09
Login to write a write a comment.
Login via GitHub