llama.cpp
py : fix missing added_tokens_dict for SPM and BPE vocabs
#4971
Merged
Overview
Commits
4
Changes
View On GitHub

py : fix missing added_tokens_dict for SPM and BPE vocabs #4971

ggerganov merged 4 commits into master from gg/fix-spm-added-tokens-dict-4958
ggerganov
ggerganov py : fix missing added_tokens_dict for SPM vocab
9b464b4e
ggerganov py : pad with unknown tokens when data is missing
a1372737
ggerganov ggerganov force pushed to a1372737 2 years ago
ggerganov py : fix BPE vocab conversion
d92351e2
ggerganov ggerganov changed the title py : fix missing added_tokens_dict for SPM vocab py : fix missing added_tokens_dict for SPM and BPE vocabs 2 years ago
ggerganov ggerganov added need feedback
TheBloke
ggerganov py : fix padded dummy tokens (I hope)
23742deb
ggerganov ggerganov merged 4f4bf35f into master 2 years ago

Login to write a write a comment.

Login via GitHub

Reviewers
No reviews
Assignees
No one assigned
Labels
need feedback
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum