transformers
b52a03cd - ⚠️⚠️[`T5Tokenize`] Fix T5 family tokenizers⚠️⚠️ (#24565)

Commit
2 years ago
⚠️⚠️[`T5Tokenize`] Fix T5 family tokenizers⚠️⚠️ (#24565) * don't add space before single letter chars that don't have a merge * fix the fix * fixup * add a test * more testing * fixup * hack to make sure fast is also fixed * update switch transformers test * revert convert slow * Update src/transformers/models/t5/tokenization_t5.py Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * add typechecking * quality --------- Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Author
Parents
Loading