llama.cpp
afd439df - unicode : include '~' in collapsed symbol class (#26972)

Commit
22 hours ago
unicode : include '~' in collapsed symbol class (#26972) The collapsed \p{S} class was missing '~', which split " ~" into separate pre-tokens and prevented the Ġ~ BPE merge used by DeepSeek V4. This caused re-tokenized prompts to diverge from sampled tokens and broke KV cache reuse. Assisted-by: Codex
Author
Parents
Loading