llama.cpp
CUDA: fuse FFN gate/up matmuls and GLU in MMQ with interleaved gate/up rows
#29948
Merged

CUDA: fuse FFN gate/up matmuls and GLU in MMQ with interleaved gate/up rows #29948

anujj
github-actions github-actions added testing
github-actions github-actions added ggml
github-actions github-actions added CUDA
anujj
anujj
JohannesGaessler
anujj anujj marked this pull request as ready for review 43 days ago
anujj anujj requested a review from ggerganov ggerganov 43 days ago
anujj anujj requested a review 43 days ago
JohannesGaessler
JohannesGaessler commented on 2026-10-08
anujj anujj force pushed to b6e0f033 42 days ago
anujj
SongXiaoXi
anujj
anujj
anujj
JohannesGaessler
JohannesGaessler commented on 2026-10-17
anujj
anujj cuda: fuse FFN gate/up and GLU in MMQ with interleaved gate/up rows
b1d280ef
anujj cuda: assert the fused MMQ gate argument and trim redundant comments
6e59326f
anujj cuda: move the fused MMQ hint check to should_fuse and add TODOs
525f87c1
anujj cuda: pass prec_src1 to the fused MMQ write-back
744d4035
anujj cuda: fuse only with efficient tiling and no fallback, merge the GLU …
4d20b62d
anujj cuda: require has_fusion explicitly, drop the fusion-only src1 precis…
ebadd88c
anujj cuda: launch the fused MMQ kernel through the stream-k kernel with on…
a11b8801
anujj cuda: inline the MMQ tiling-efficiency check
30840769
anujj tests: one fused gate/up MMQ case per type and GLU op
352d5106
anujj cuda: always launch the fused MMQ kernel with one block per tile
7d9a32d4
anujj tests: shorten the fused gate/up MMQ test comment
6e34e56d
anujj anujj force pushed to 6e34e56d 30 days ago
anujj
JohannesGaessler
JohannesGaessler commented on 2026-10-23
anujj cuda: select the fused MMQ loader and write-back in ggml_cuda_mmq_get…
5d4638a5
anujj cuda: run the fused MMQ kernel on the plain tiling path
8a357f4f
anujj cuda: put the column tiles innermost in the MMQ plain tiling path
d63ccd4f
JohannesGaessler
JohannesGaessler approved these changes on 2026-10-24
JohannesGaessler
JohannesGaessler
JohannesGaessler JohannesGaessler added merge ready
ggerganov ggerganov merged c49edf09 into master 26 days ago

Login to write a write a comment.

Login via GitHub

Assignees
No one assigned
Labels
Milestone