llama.cpp
0fb8f157 - Title: CUDA: few-row MMQ for BS 2-24 (#31533)

Commit
1 day ago
Title: CUDA: few-row MMQ for BS 2-24 (#31533) * CUDA: few-row MMQ for 2-24 src1 columns New int8 MMA kernel for quantized mat-muls with 2-24 src1 columns, e.g. speculative decoding verify passes. Supports Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ3_S, IQ4_NL and IQ4_XS on NVIDIA GPUs from Ampere on. * nit fixes * CUDA-only guard for few-row MMQ, weights flag in test_mul_mat * few-row quantize: DS4 sum from the quantized values, as in #31247 * few-row MMQ: clearer HIP/MUSA abort, test_mul_mat two-context build * few-row MMQ: dispatch order MMVQ, few-row, MMQ
Author
Parents
Loading