transformers
de01a22a - Fix edge case for tokenize (#36277) (#36555)

Commit
344 days ago
Fix edge case for tokenize (#36277) (#36555) * Fix edge case for tokenize (#36277) * Fix tokenizing dtype for float input cases * add test for empty input string * deal empty list of list like [[]] * add tests for tokenizer for models with input that is not plain text
Author
Parents
Loading