llama.cpp
ad277572 - Move to backend sampling for MTP draft path (#23287)

Commit
68 days ago
Move to backend sampling for MTP draft path (#23287) * Move to backend sampling for MTP draft path Run top_k(10) on the draft backend. D2H transfers happen only for the top 10 logits Make backend sampling more robust and fallback to CPU on failure cases, such as with "-sm tensor" or when a backend doesn't support TOP_K. * Allow sampler chains to be partially offloaded to backend * Add --spec-draft-backend-sampling argument. Enabled by default.
Author
Parents
Loading