llama.cpp
aaa807c5 - cuda: decode the conv2d patch indices with fastdiv (#32492)

Commit
1 day ago
cuda: decode the conv2d patch indices with fastdiv (#32492) The implicit GEMM conv2d kernel recovers the channel and the kernel position of each reduction index with three integer divisions by runtime divisors, once per loaded row of every tile. Outside the 3x3 and 1x1 specializations this integer division bounds the kernel: removing it halves the time on long reductions. Divide with the precomputed magic numbers of fastdiv instead. The 3x3 and 1x1 specializations keep their compile time divisors. test-backend-ops -o CONV_2D: 2006/2006 on CUDA0. CONV_2D, F16 kernel, us/run on an RTX PRO 6000: [2054,1,768] k [7,1,768,768] 221 -> 138 [16390,1,384] k [7,1,384,384] 433 -> 266 [65542,1,192] k [7,1,192,192] 420 -> 271 [8240,1,256] k [11,1,256,256] 165 -> 108 [19,19,256,16] k [4,4,256,4096] 1672 -> 1076
Parents
Loading