llama.cpp
41d8297f - metal : CONV_2D tensor API for M5, split-K for all chips (#32542)

Commit
1 day ago
metal : CONV_2D tensor API for M5, split-K for all chips (#32542) * metal: add a tensor API conv_2d kernel and split long reductions The conv_2d matmul kernel runs on simdgroup matrices only, and gives each threadgroup the whole reduction of its output tile. On M5 this leaves the tensor cores unused, and convolutions with few output positions and a long reduction (IC * KW * KH in the thousands) run a handful of threadgroups on an otherwise idle GPU. Add kernel_conv_2d_mm_tensor for devices with the tensor API: like kernel_mul_mm, it gathers the patches of 64 output positions into threadgroup memory as the left operand and reads the kernel rows from device memory as the right operand. Devices without the tensor API keep the simdgroup kernel. Both kernels can now split the reduction until there are enough threadgroups: each split writes its partial sums to a buffer placed after the output, as flash attention does, and a small kernel adds them in a fixed order, so the result stays deterministic. CONV_2D is marked as an op whose allocation may exceed its output size. test-backend-ops -o CONV_2D passes, with and without GGML_METAL_TENSOR_DISABLE. CONV_2D, F16 kernel, us/run on an M5 Max: [19,19,256,16] k [4,4,256,4096] 19008 -> 5235 [16,16,128,8] k [3,3,128,512] 333 -> 124 [1536,1536,32,1] k [3,3,32,64] 12086 -> 6512 [33,1,1280,1] k [3,1,1280,1280] s2 659 -> 33 [2054,1,768,1] k [7,1,768,768] 3161 -> 699 * metal: walk the conv_2d patches incrementally, size splits by output The patch gather of both conv_2d matmul kernels spent most of its time on address math: 64-bit multiplies, divisions and four bound checks per value. Walk the reduction index with 32-bit byte offsets inside the image, updated by additions, with one unsigned compare per bound; the host checks that an image fits. The tensor kernel no longer gathers the reduction indices past the end of its split. The number of splits now targets about 1M partial sums, with at least 3 reduction steps per split, instead of counting the tiles of one kernel. CONV_2D, F16 kernel, us/run on an M5 Max: [19,19,256,16] k [4,4,256,4096] 4998 -> 4512 [1536,1536,32,1] k [3,3,32,64] 6721 -> 5354 [4096,1,1024,1] k [7,1,1024,1024] 2701 -> 2236 [2054,1,768,1] k [7,1,768,768] 1050 -> 871 with GGML_METAL_TENSOR_DISABLE: [4096,1,1024,1] k [7,1,1024,1024] 9047 -> 8283 * nit: share the CONV_2D simdgroup count between host and kernel kernel_conv_2d_mm and its pipeline both hardcoded 4 simdgroups per threadgroup. N_SG_CONV_2D in ggml-metal-impl.h now feeds the kernel constant and res.nsg, so the dispatch and the kernel cannot drift apart. * metal: fall back to the generic conv_2d kernel for images past 4 GiB The matmul conv_2d kernels walk the patches with 32-bit byte offsets inside an image. Instead of asserting on nb13, compute the largest offset actually read from ne/nb, which also covers strided views of the input, and route images that do not fit to the generic kernel. * cont : comments Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Parents
Loading