transformers
8a5a4b5c - gguf user defined tokens (#49008)

Commit
39 days ago
gguf user defined tokens (#49008) * GGUF tokenizer: keep the file's user-defined tokens whole A GGUF marks tokens like `<think>`, `</think>`, `<tool_call>` and `<tool_response>` as USER_DEFINED. llama.cpp always matches these whole in the text, and the reference tokenizers carry them as non-special added tokens. The GGUF-built tokenizer only registered CONTROL tokens, so `<think>` was cut into `<`, `think`, `>`: every chat prompt for Qwen3.5 differed from what llama.cpp feeds the model. Add them as plain added tokens, restricted to spellings already in the vocabulary so a sentencepiece file's raw whitespace runs do not grow it. * update comment
Author
Parents
Loading