llama.cpp

fccf7166 - HIP: broaden MoE ncols_opt tile heuristic on RDNA3.5 architecture (#28935)

HIP: broaden MoE ncols_opt tile heuristic on RDNA3.5 architecture (#28935) It's found the MoE ncols_opt tile heuristic needs to be broadened to include the RDNA3.5 architecture. The code change is implemented in ggml/src/ggml-cuda/mmq.cu and just change the GGML_CUDA_CC_IS_RDNA3_0 to GGML_CUDA_CC_IS_RDNA3 in the condition. The dense dispatch logic remains unchanged. The Test machine configuration we used is AMD Radeon 8060S, gfx1151 (RDNA3.5), 20 CU, wave32 + AMD Ryzen AI MAX+ 388, 8C/16T, 23.79 GB RAM we complete the Correctness verification and performance evaluation as follows: test-backend-ops test -b ROCm0 -o MUL_MAT -p type_a=<q4_K|q5_K|q4_0|q5_0> test-backend-ops test -b ROCm0 -o MUL_MAT_ID -p type_a=<q4_K|q5_K|q4_0|q5_0> all pass: MUL_MAT 64/64, 29/29, 48/48, 14/14; MUL_MAT_ID 84/84, 3/3, 74/74, 3/3 Performance result on target machine: LFM2.5-8B-A1B-UD-Q4_K_M (Q4_K MoE) +16.198% [+12.704, +19.799] 8/8 Qwen1.5-MoE-A2.7B-Q2_K (Q2_K MoE) +6.189% [ +5.245, +7.141] 8/8 pooled (16 pairs) +11.081% [ +7.972, +14.279] 16/16 Token generation (tg128) is unchanged on the Q4_K MoE model and +2.188% [+0.905, +3.488] on the Q2_K one.