server : support unified cache across slots #16736
ggerganov
merged
14 commits into
master
from
gg/server-unified-slots
slaren
commented
on 2025-10-23
ggerganov
commented
on 2025-10-29
slaren
commented
on 2025-11-01
server : support unified context across slots
57ece5ba
cont : fix speculative decoding initialization
a42fb771
context : fix n_ctx_per_seq computation
492f628c
server : purge slots one by one
8222e9c2
tests : add unified cache server tests
21791750
llama : update per-seq context computation
f0f105ff
test-thread-safety : handle tiny training context of the input model
e7b7cbfb
server : fix server_tokens clear()
290f6a9f
server : use 4 slots + unified KV by default
23323cd1
llama : add note about context size queries
f2cca024
cont : update todos [no ci]
ff684363
context : do not cap the size of the context
c08d0d14
tests : adjust parameters to be CI friendlier
356dc08b
slaren
approved these changes
on 2025-11-01
context : add warning
56fceee2
Login to write a write a comment.
Login via GitHub