llama.cpp
d4f5c260 - CUDA: use 64-bit dst offsets in mul_mat_q (#32374)

Commit
1 day ago
CUDA: use 64-bit dst offsets in mul_mat_q (#32374) The MMQ kernels computed the dst tile offset (jt*J*stride_col_dst) in 32-bit int. For a dst with more than 2^31 elements, column tiles that start past INT_MAX/nrows wrap around and write out of bounds. This happens with an all-logits output head, e.g. llama-perplexity -c 32768 -b 32768 -ub 16384 on a 248320-vocab model: the q8_0 output matmul has a 248320 x 16384 dst (4.07e9 elements) and aborts with an illegal memory access. Widen the offsets in mul_mat_q and the stream-k fixup to int64_t, and add a (disabled by default, ~9 GB) test-backend-ops case. Assisted-by: Claude Opus 5.5
Author
Parents
Loading