convert : refactor vocab selection logic #6355
cebtenzzre
merged
13 commits into
master
from
ceb/fix-convert-bpe-hf
convert : remove redundant annotations
dd1a60c5
convert : remove unused vocab attributes
72e95e33
convert : vocab inheritance instead of duck typing
9803bb72
convert-persimmon : typing fixup
03f0c2e8
convert : do not allow "no_vocab" in --vocab-types
d852c61d
convert : use context managers with most file handles
b2b63d13
convert : fix incorrect added token dedup in BpeVocab
d12a63ca
convert : use appropriate exception types
8d2ac2cc
convert-hf : fix type of tokens after #3252
2e6fd63b
convert : refactor vocab selection logic
79852ab8
convert-hf : HfVocab -> LlamaHfVocab
ebad773e
llama : update vocab type descriptions to reflect actual meaning
80e9fc7c
ggerganov
approved these changes
on 2024-03-28
convert : appease flake8
d09e4ac6
Login to write a write a comment.
Login via GitHub