llama.cpp
CUDA: branchless Q4_K/Q5_K unpack to speed up mmvq, L2 prefetch on DGX Spark
#26705

Merged

CUDA: branchless Q4_K/Q5_K unpack to speed up mmvq, L2 prefetch on DGX Spark #26705

am17an merged 10 commits into ggml-org:master from praneshgo:pgonegandla/mmvq-q4k-q5k-branchless

github-actions added ggml

github-actions added CUDA

praneshgo marked this pull request as ready for review 58 days ago

praneshgo requested a review 58 days ago

praneshgo force pushed 51 days ago

praneshgo changed the title ~~CUDA: Update Q4_K and Q5_K to use branchless computation, which stops the scale unpack being re-executed for every column in mmvq, improving perf at batch sizes > 1~~ CUDA: Adding Spark to use prefetch and update Q4_K and Q5_K to use branchless computation, which stops the scale unpack being re-executed for every column in mmvq, improving perf at batch sizes > 1 50 days ago

gaugarg-nv commented on 2026-08-31

Update Q4_K and Q5_K to use branchless computation, which stops the s…

10d708ac

Gating the change off from DGX Spark due to no gain

b6868489

Adding prefetch gated to Spark, making branchless change in Q4_K and …

6754ceef

Guard the mmvq L2 prefetch against MUSA as well as HIP

9b9cf680

gaugarg-nv commented on 2026-08-31

Define the mmvq L2 prefetch only under the Spark guard

63daa18b

praneshgo force pushed to 63daa18b 46 days ago

Update switch point for Q4_K to accommodate more models

99e1393d

IMbackK assigned

IMbackK 43 days ago

IMbackK commented on 2026-09-03

Remove stale comments

2097c8cd

gaugarg-nv approved these changes on 2026-09-04

IMbackK unassigned

IMbackK 42 days ago

am17an commented on 2026-09-07

Add block_size to ggml_cuda_type_traits and create a separate mmvq_sh…

9c2afe6c

am17an commented on 2026-09-07

am17an approved these changes on 2026-09-07

Rename block_size to bs for cleaner indentation

22c832a5

gaugarg-nv approved these changes on 2026-09-07

Fix build error on non-Spark CUDA arch with appropriate conditional a…

aa17a1bc

praneshgo changed the title ~~CUDA: Adding Spark to use prefetch and update Q4_K and Q5_K to use branchless computation, which stops the scale unpack being re-executed for every column in mmvq, improving perf at batch sizes > 1~~ CUDA: branchless Q4_K/Q5_K unpack to speed up mmvq, L2 prefetch on DGX Spark 39 days ago

am17an approved these changes on 2026-09-07

am17an merged 73ab7599 into master 39 days ago

Reviewers

am17an

gaugarg-nv

IMbackK

Assignees

No one assigned

Labels

ggml CUDA

Milestone

No milestone

llama.cpp CUDA: branchless Q4_K/Q5_K unpack to speed up mmvq, L2 prefetch on DGX Spark #26705 Merged

CUDA: branchless Q4_K/Q5_K unpack to speed up mmvq, L2 prefetch on DGX Spark #26705

llama.cpp
CUDA: branchless Q4_K/Q5_K unpack to speed up mmvq, L2 prefetch on DGX Spark
#26705

Merged