langchain
f435f226 - Use tiktoken for Python 3.8 (#2709)

Commit
3 years ago
Use tiktoken for Python 3.8 (#2709) Fixes issue #2677 `tiktoken` is supported for Python 3.8, so there is no need to use the fallback GPT-2 tokenizer.
Author
Parents
Loading