transformers
260c8608 - Merge pull request #1027 from samvelyan/iterative_split_on_token

Commit
6 years ago
Merge pull request #1027 from samvelyan/iterative_split_on_token Re-implemented tokenize() iteratively in PreTrainedTokenizer.
Author
Loading