transformers
1c7c34bc
- Improve `PreTrainedTokenizerFast` loading time when there are many added tokens (#31404)
Go
Login via GitHub
Home
Pricing
FAQ
Install
Login
via GitHub
Commit
View On
GitHub
Commit
2 years ago
Improve `PreTrainedTokenizerFast` loading time when there are many added tokens (#31404) * use hash * use hash * update --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
References
#31404 - Improve `PreTrainedTokenizerFast` loading time when there are many added tokens
Author
ydshieh
Parents
6e56b834
Loading