langchain
f435f226
- Use tiktoken for Python 3.8 (#2709)
Go
Login via GitHub
Home
Pricing
FAQ
Install
Login
via GitHub
Commit
View On
GitHub
Commit
3 years ago
Use tiktoken for Python 3.8 (#2709) Fixes issue #2677 `tiktoken` is supported for Python 3.8, so there is no need to use the fallback GPT-2 tokenizer.
References
#2709 - Use tiktoken for Python 3.8
Author
joshsny
Parents
186ca9d3
Loading