llama.cpp
Add byte token type when tokenizer.model is not exists
#4641

Merged

Add byte token type when tokenizer.model is not exists #4641

strutive07 merged 2 commits into ggml-org:master from strutive07:feature/byte_token_convert

Add byte token type to hf format

9f297f81

remove unused variable

ff8b9941

strutive07 assigned

strutive07 2 years ago

slaren approved these changes on 2023-12-26

strutive07 merged f56d6077 into master 2 years ago

Reviewers

slaren

Assignees

strutive07

Labels

None yet

Milestone

No milestone