[CB] Add a "max_requests_per_batch" parameter #46434
remi-or
merged
23 commits into
main
from
cb-max-requests
Broaden the safety margin and add it to the config
b8015ba2
Plumbing
834e71cd
IO take attributes from the CB config
3eb4da94
Cap max req per batch to max tokens per batch
3832f784
Update cfg is now a fn
507ffdae
Finalize to model runner and IO
c374a0ae
Rename to be grammaticaly correct
1bbc9445
Auto review
c19b2194
Small fixes
8b18d4d3
Fix imports and remove debub print
4b8716c1
Fix wrong max value for padding
a67dfb22
Solved an aleady solved TODO
7e832096
Add progress bar now that there is less info
8c6a0a9c
Typos
edb22b35
Add a warning for persistent + sized manager
abe1f2c7
Add TODO
cb8f8c24
Fix wrong bound
5c51d943
Fix logit processors
f655091b
Fix eager crash of FA2
c1687725
Test an edge case
c629d46d
ArthurZucker
approved these changes
on 2026-06-09
Better comment
39615646
Merge branch 'main' into cb-max-requests
86623d1b
Merge branch 'main' into cb-max-requests
37df29a3
Login to write a write a comment.
Login via GitHub