llama.cpp
2960eb29 - vulkan: Use one row per workgroup for f32 mmv (#17711)

Commit
24 days ago
vulkan: Use one row per workgroup for f32 mmv (#17711) The MoE models have a mul_mat_vec with very small m (32, 64, 128) right before the topk_moe selection. Running multiple rows per wg doesn't utilize the SMs well. I think even for larger m, f32 is so bandwidth-limited that running multiple rows doesn't help.
Author
Parents
Loading