llama.cpp
3fb47677 - common : add --spec-draft-n-ubatch (#30459)

Commit
2 days ago
common : add --spec-draft-n-ubatch (#30459) * common : add --spec-draft-n-ubatch Allow configuring the physical batch size of the draft context. The drafters feed it at most the drafted tokens per sequence and chunk larger batches by its own n_ubatch, so it does not need the ubatch of the target. A smaller value shrinks the draft compute buffer and leaves more VRAM for the target KV cache (the draft is fitted together with the target). The default 0 keeps the current behavior of following --n-ubatch. Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next * cont : remove comment [no ci]
Author
Parents
Loading