llama.cpp
9ac2693a - server: fix n_cmpl not skipping processing prompt (#18663)

Commit
8 days ago
server: fix n_cmpl not skipping processing prompt (#18663) * server: fix n_cmpl not skipping processing * fix infinite loop on empty batch * cont : init child samplers + modify child logic * cont : cleanup * cont : improve n_cmpl logic - launch the parent task first so it finds the slot with best cache - parent task waits for child tasks to be launched - when a child task finishes - remove its cache * cont : remove redundant function * cont : reduce parent checks * fix : nullptr task dereference --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Author
Parents
Loading