transformers
fix: set `clean_up_tokenization_spaces=False` in Llama 3 tokenizer conversion
#44914
Merged
Merged
fix: set `clean_up_tokenization_spaces=False` in Llama 3 tokenizer conversion #44914
ArthurZucker
merged
1 commit into
huggingface:main
from
maxsloef-goodfire:fix/llama3-clean-up-tokenization-spaces
fix: set clean_up_tokenization_spaces=False in Llama 3 tokenizer conv…
8969dde2
ArthurZucker
commented
on 2026-03-23
Login to write a write a comment.
Login via GitHub