llama.cpp
CUDA: Improve flash decoding kernel GPU occupancy for BS=1 case
#12183

Merged

CUDA: Improve flash decoding kernel GPU occupancy for BS=1 case #12183

IMbackK merged 3 commits into ggml-org:master from gaugarg-nv:flash_decoding_improvement

gaugarg-nv requested a review from

JohannesGaessler 285 days ago

github-actions added Nvidia GPU

github-actions added ggml

gaugarg-nv force pushed 285 days ago

gaugarg-nv changed the title ~~CUDA: Improve flash decoding kernel occupancy for BS=1 case~~ CUDA: Improve flash decoding kernel GPU occupancy for BS=1 case 284 days ago

ggerganov commented on 2025-03-06

JohannesGaessler commented on 2025-03-06

gaugarg-nv force pushed 282 days ago

github-actions added testing

gaugarg-nv force pushed to b6e067b9 273 days ago

CUDA: determine FA parallel blocks at runtime

2d011e65

CUDA: Improve flash decoding kernel occupancy for BS=1 case

aa5aa01d

gaugarg-nv force pushed from b6e067b9 to aa5aa01d 270 days ago

consider tail effects for parallel_blocks

66d873b6

JohannesGaessler approved these changes on 2025-03-19

IMbackK approved these changes on 2025-03-19

ggerganov approved these changes on 2025-03-19

IMbackK merged 517b5ddb into master 270 days ago

gaugarg-nv deleted the flash_decoding_improvement branch 269 days ago

Reviewers

ggerganov

JohannesGaessler

IMbackK

Assignees

No one assigned

Labels

testing Nvidia GPU ggml

Milestone

No milestone

llama.cpp CUDA: Improve flash decoding kernel GPU occupancy for BS=1 case #12183 Merged

CUDA: Improve flash decoding kernel GPU occupancy for BS=1 case #12183

llama.cpp
CUDA: Improve flash decoding kernel GPU occupancy for BS=1 case
#12183

Merged