⚠️⚠️[`T5Tokenize`] Fix T5 family tokenizers⚠️⚠️ #24565
ArthurZucker
merged
12 commits into
huggingface:main
from
ArthurZucker:fix-t5-tokenizer
don't add space before single letter chars that don't have a merge
e03a7685
fix the fix
76d6ab39
fixup
5a7184bb
add a test
baac7be9
more testing
6e37601e
fixup
b9333287
hack to make sure fast is also fixed
d0cbc495
update switch transformers test
50008ed2
revert convert slow
5edf8633
Update src/transformers/models/t5/tokenization_t5.py
17bda2cd
add typechecking
059999e1
quality
8d3f2a2f
Ethan-yt
commented
on 2023-08-04
Login to write a write a comment.
Login via GitHub