transformers
deceb001 - Merge pull request #2177 from mandubian/issue-2106

Commit

6 years ago

Merge pull request #2177 from mandubian/issue-2106 :zip: #2106 tokenizer.tokenize speed improvement (3-8x) by caching added_tokens in a Set

References

Author

thomwolf

Parents