transformers
[`tokenizers`] Ensure that add_prefix_space is propagated to backend_tokenizer.pre_tokenizer
#35593
Merged
Merged
[`tokenizers`] Ensure that add_prefix_space is propagated to backend_tokenizer.pre_tokenizer #35593
tomaarsen
merged
4 commits into
huggingface:main
from
tomaarsen:tokenization/add_prefix_space_patch
Ensure that add_prefix_space is propagated to backend_tokenizer.pre_t…
e8738e5d
Simplify setting self.add_prefix_space, ensure pre_tok exists
8df9846a
Wrap in try-except to catch 'Custom PreTokenizer cannot be serialized'
aaffcb47
Propagate add_prefix_space in T5TokenizerFast to superclass
9888d790
ArthurZucker
approved these changes
on 2025-01-09
Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub