kv-cache : refactor + add llama_memory_state_i #13746
ggerganov
merged
14 commits into
master
from
gg/kv-cache-simplify-part3
Base automatically changed from
gg/kv-cache-simplify-part2
to
master
1 year ago
ggerganov
commented
on 2025-05-25
ggerganov
commented
on 2025-05-25
gabe-l-hart
commented
on 2025-05-27
gabe-l-hart
commented
on 2025-05-27
gabe-l-hart
commented
on 2025-05-28
gabe-l-hart
commented
on 2025-05-28
slaren
approved these changes
on 2025-05-29
kv-cache : simplify the "struct llama_kv_cache" interface
773b6e39
kv-cache : revert the (n_swa + n_ubatch) change (for next PR)
9fc50dcd
kv-cache : some comments
c2c35917
context : fix graph reserve for multiple sequences
88567820
kv-cache : fix typo [no ci]
bffb9d4a
kv-cache : fix find_slot() logic for free slots
32cc9eab
llama : add TODO for deprecating the defrag API in the future
f97de9b7
kv-cache : improve find_slot() using min/max seq pos info
7764d914
llama : handle aborts and compute errors
780bba94
memory : extract state into llama_memory_state
dbcfa5f1
kv-cache : add comments
f2ded9d4
server : update batching logic to reset n_batch on successful decode
e230e514
server : upon full re-processing, remove the sequence from the cache
3cf51863
kv-cache : add TODO for doing split_equal when split_simple fails
71619f2d
ggerganov
changed the title
Reviewers
Assignees
No one assigned
Labels
examples
server
Milestone
No milestone
Login to write a write a comment.
Login via GitHub