llama.cpp
fix(gguf-py): special tokens are no longer skipped when add_<token>_token is set to false
#5487

Merged

fix(gguf-py): special tokens are no longer skipped when add_<token>_token is set to false #5487

slaren merged 2 commits into ggml-org:master from vriesdemichael:fix/5040-eos-bos-token-missing-in-gguf

fix(gguf-py): special tokens are no longer skipped when add_<token>_t…

f428652b

ggerganov commented on 2024-02-15

vriesdemichael force pushed from de5ff227 to b6564234 1 year ago

vriesdemichael force pushed from b6564234 to 10f96516 1 year ago

ggerganov approved these changes on 2024-02-15

ggerganov requested a review from

slaren 1 year ago

fix(gguf-py): added missing cls and mask token ids to the gguf metadata

3b5dc11b

vriesdemichael force pushed from 10f96516 to 3b5dc11b 1 year ago

slaren approved these changes on 2024-02-15

slaren merged 73122473 into master 1 year ago

Reviewers

slaren

ggerganov

Assignees

No one assigned

Labels

None yet

Milestone

No milestone