transformers
[serving] Fix continuous batching JSON response serialization
#45057
Merged
Merged
[serving] Fix continuous batching JSON response serialization #45057
NathanHB
merged
10 commits into
huggingface:main
from
NathanHB:fix-continuous-batching-json-response
Fix continuous batching JSON response serialization
b126ffdc
LysandreJik
commented
on 2026-03-27
add example script eval-job
1a9bee08
fix script
5ee9fcff
Add test for continuous batching non-streaming JSON response
f86120cc
fix ci
92899701
Update eval script to use official transformers repo main branch
5fddeb2a
ArthurZucker
approved these changes
on 2026-03-30
add kernels and flash attn 2
db4a7746
Add continuous batching configuration CLI arguments to serve command
e5fd8cc0
Add thread lock for manager creation to avoid double manager
f9729fa6
LysandreJik
approved these changes
on 2026-03-31
change transformers dep
9f7e1841
Login to write a write a comment.
Login via GitHub