llama.cpp
CUDA: Improve performance via less synchronizations between token
#17795
Merged
Merged
CUDA: Improve performance via less synchronizations between token #17795
ggerganov
merged
13 commits into
ggml-org:master
from
aendk:akieslinger/reduce-per-token-syncs
JohannesGaessler
commented
on 2025-12-08
aendk
changed the title 
ggerganov
commented
on 2026-01-15
Adds CPU-to-CUDA copy capability to
d5e3b24a
Adds function to relax sync requirements between input copies on
76deb1f1
Exchanges synchronous copy with async copy function.
2b4b80d0
Adds macro guards to allow compilation in non-CUDA builds
d8cebf68
Reworked backend detection in ggml-backend.cpp to avoid linking
51e8cc79
Relax requirement of checks in async CUDA copies from backend and buf…
e51e58fe
Minor cleanup
1419050d
Makes opt-in to relax use of explicit syncs more general. Backends like
353daedd
Reintroduces stricter check for CPU->CUDA backend async copy via
34cf1027
Corrects initialization of ggml_backend_sync_mode in
c3998dc0
Simplifies synchronizations to adhere to `saaasg` pattern.
d58d96b7
Apply suggestion from @ggerganov (src->buffer to buf_src)
887c2fb0
Apply suggestion from @ggerganov (src->buffer to buf_src) v2
81c96185
ggerganov
approved these changes
on 2026-03-05
Reviewers
Assignees
No one assigned
Labels
Nvidia GPU
ggml
Milestone
No milestone
Login to write a write a comment.
Login via GitHub