transformers
cc013513 - :zip: #2106 basic tokenizer.tokenize global speed improvement (3-8x) by simply caching added_tokens in a Set

Commit
6 years ago
:zip: #2106 basic tokenizer.tokenize global speed improvement (3-8x) by simply caching added_tokens in a Set
Author
Pascal Voitot
Committer
Pascal Voitot
Parents
Loading