transformers
260c8608
- Merge pull request #1027 from samvelyan/iterative_split_on_token
Go
Login via GitHub
Home
Pricing
FAQ
Install
Login
via GitHub
Commit
View On
GitHub
Commit
6 years ago
Merge pull request #1027 from samvelyan/iterative_split_on_token Re-implemented tokenize() iteratively in PreTrainedTokenizer.
References
#1027 - Re-implemented tokenize() iteratively in PreTrainedTokenizer.
Author
thomwolf
Parents
9beaa85b
d30cbaf5
Loading