llama.cpp
decaf508 - server: refactor + correctness fixes for metrics (#26920)

Commit
54 days ago
server: refactor + correctness fixes for metrics (#26920) * server: refactor metrics * move most fields to server_slot_stats * cont * rm result_timings * tie stats to batch * cont * nits: move place in code * exclude first generated token * more accurate batch metrics tracking * n_predict --> n_gen * metrics_on_prediction * metrics_flush_idle * metrics: seperate cache/processed prompt tokens * refactor server_task_result_metrics * add test * nits * fix flush before reset() * cont * rm dead code * nits
Author
Parents
Loading