llama.cpp
18443257 - server: add ctx-per-slot (--kv-unified-per-slot) (#24124)

Commit
55 days ago
server: add ctx-per-slot (--kv-unified-per-slot) (#24124) * Add ctx-per-slot argument for unifid KV cache * Swap out ctx fractions for ctx pool slots * Formatting cleanup * Remove ctx-pool-slots, make ctx-per-slot an int * refactor it --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Author
Parents
Loading