llama.cpp
de6f727a - llama: limit max outputs of `llama_context` (#23861)

Commit
92 days ago
llama: limit max outputs of `llama_context` (#23861) * llama: save more VRAM by reserving n_outputs == n_seqs when possible * add n_outputs_per_seq * move n_outputs_max to server-context * change ubatch to batch everywhere
Author
Parents
Loading