llama : add high-throughput mode #14363
ggerganov
merged
17 commits into
master
from
gg/llama-high-throughput
Base automatically changed from
gg/kv-cache-use-set-rows
to
master
198 days ago
kv-cache : prepare K/V buffers for separation
be82648b
batched-bench : fix oob write
5a354755
llama : add "virtual sequences"
45ecf841
llama : use "stream" vs "virtual sequence"
4c2d6510
graph : fix stream splitting when KV cache is not used
0d05acd6
kv-cache : add multi-stream save/load support
247015ee
llama : add "--attn-streams" flag
3354ce7e
kv-cache : fix handling when find_slot fails
18fb95dd
kv-cache : restore find_slot impl
cbe971ae
kv-cache : add comments
1b4fbc8f
kv-cache : add bounds checks for sequence id
8bf7fec0
cont : add n_seq_max to batch allocr
91751ead
kv-cache : perform stream copies lazily after llama_synchronize
2d08a395
kv-cache : avoid throwing exceptions across the C boundary
69169b15
CUDA: 4D FlashAttention support (#14628)
886d3f15
llama : rename attn_streams -> kv_unified
fb8150d8
slaren
approved these changes
on 2025-07-16
common : rename kv_split -> kv_unified
318c4f8f
Reviewers
Assignees
No one assigned
Labels
testing
Nvidia GPU
examples
ggml
Apple Metal
hot
Milestone
No milestone
Login to write a write a comment.
Login via GitHub