llama.cpp
Improve handling of special tokens in GGML to GGUF converter
#2725
Merged
Merged
Improve handling of special tokens in GGML to GGUF converter #2725
KerfuffleV2
merged
3 commits into
ggml-org:master
from
KerfuffleV2:feat-improve-ggml-to-gguf-special-tokens
Improve UNK, BOS, EOS token handling when converting without metadata.
14ed02e8
Set default UNK token mapping from -1 to 0 in llama.cpp
bb58495e
Try to handle overflow due to buggy Windows Python with a better erro…
f382606a
KerfuffleV2
changed the title 
klosax
approved these changes
on 2023-08-22
Login to write a write a comment.
Login via GitHub