transformers
[serving] Fix continuous batching JSON response serialization
#45057

Merged

Commits

Fix continuous batching JSON response serialization

NathanHB committed 14 days ago
add example script eval-job

NathanHB committed 13 days ago
fix script

NathanHB committed 13 days ago
Add test for continuous batching non-streaming JSON response

NathanHB committed 13 days ago
fix ci

NathanHB committed 13 days ago
Update eval script to use official transformers repo main branch

NathanHB committed 13 days ago
add kernels and flash attn 2

NathanHB committed 10 days ago
Add continuous batching configuration CLI arguments to serve command

NathanHB committed 10 days ago
Add thread lock for manager creation to avoid double manager

remi-or committed 9 days ago
change transformers dep

NathanHB committed 9 days ago