text-generation-inference
Expose the real-time internal state of the batcher through SSE
#3065
Open
Open
Expose the real-time internal state of the batcher through SSE #3065
mfuntowicz
wants to merge
5 commits into
main
from
proxy_sse_engine_state
feat(metrics): exposes queue size as tokens along with individual req…
bb8f5963
feat(metrics): exposes the engine state as an endpoint
8de41f63
feat(metrics): update Cargo.lock
1a9c5dec
feat(metrics): dispatch internal engine state event from queuing/batc…
712199c7
feat(metrics): remove ngrok mandatory feature for backendv3 crate
f72547c9
Login to write a write a comment.
Login via GitHub