llama.cpp
91f64993
- Respect tokenizer.ggml.add_bos_token value when tokenizing (#4040)
Go
Login via GitHub
Home
Pricing
FAQ
Install
Login
via GitHub
Commit
View On
GitHub
Commit
2 years ago
Respect tokenizer.ggml.add_bos_token value when tokenizing (#4040) * gguf-py: gguf-dump: Respect --no-tensor flag in JSON mode. * Respect add_bos_token GGUF metadata value * gguf-py: Try to fix SpecialVocab giving up too easily for the Nth time
References
#4040 - Respect tokenizer.ggml.add_bos_token value when tokenizing
Author
KerfuffleV2
Parents
8da46278
Loading