Changed convert-pt-to-ggml.py to use .tiktoken tokenizer files #725
Changed convert-pt-to-ggml to use .tiktoken tokenizer files
c0ad8bb8
ivan-gorin
force pushed
from
cda97bc1
to
c0ad8bb8
3 years ago
ggerganov
approved these changes
on 2023-04-14
ggerganov
merged
62b51c30
into master 3 years ago
Assignees
No one assigned
Login to write a write a comment.
Login via GitHub