llama.cpp
a518119d - llama: keep the backend sampling graph static across ubatches (#30223)

Commit
24 days ago
llama: keep the backend sampling graph static across ubatches (#30223) The reserve builds n_outputs_max_per_seq sampling chains per sampler, while a decode built one per output row, so the graph changed its topology after the reserve and GGML_SCHED_NO_REALLOC builds aborted on the next same sized graph. Every sampler now builds n_outputs_max_per_seq chains, the ones without a row of the ubatch on the padding row and not selected, and graph_max_nodes counts them.
Parents
Loading