llama.cpp
cc31369f - metal: add a tensor API conv_2d kernel and split long reductions

Commit
1 day ago
metal: add a tensor API conv_2d kernel and split long reductions The conv_2d matmul kernel runs on simdgroup matrices only, and gives each threadgroup the whole reduction of its output tile. On M5 this leaves the tensor cores unused, and convolutions with few output positions and a long reduction (IC * KW * KH in the thousands) run a handful of threadgroups on an otherwise idle GPU. Add kernel_conv_2d_mm_tensor for devices with the tensor API: like kernel_mul_mm, it gathers the patches of 64 output positions into threadgroup memory as the left operand and reads the kernel rows from device memory as the right operand. Devices without the tensor API keep the simdgroup kernel. Both kernels can now split the reduction until there are enough threadgroups: each split writes its partial sums to a buffer placed after the output, as flash attention does, and a small kernel adds them in a fixed order, so the result stays deterministic. CONV_2D is marked as an op whose allocation may exceed its output size. test-backend-ops -o CONV_2D passes, with and without GGML_METAL_TENSOR_DISABLE. CONV_2D, F16 kernel, us/run on an M5 Max: [19,19,256,16] k [4,4,256,4096] 19008 -> 5235 [16,16,128,8] k [3,3,128,512] 333 -> 124 [1536,1536,32,1] k [3,3,32,64] 12086 -> 6512 [33,1,1280,1] k [3,1,1280,1280] s2 659 -> 33 [2054,1,768,1] k [7,1,768,768] 3161 -> 699
Committer
Parents
Loading