[CB] [Major] Asynchronous batching #43960
ArthurZucker
merged
39 commits into
main
from
cb-async-lfg
Cleanup: batch is more self contained
334afd38
Created utils.py file
c8d7753f
Moved pad to utils
8aee274d
Pin memory for input and outputs
3bc03161
Consolidate inputs into a bulk tensor
1bcc5264
Consolidated read and write indices
9a84bde1
Add the transfer_inputs fn
92947aa6
Renames and getters
a258ba12
Remove useless sync
c5877bc7
Move graphs to the IOs
a75124c8
Async done except for carry_in_ids
b4c13fdf
Add carry over (scheduler not picking up tho)
da98e66e
Remodeled scheduling
b736a136
Fix carry over
462ff769
Fix stream
810b55f1
Bumped _upper_bound_num_blocks
8ead282e
Faster compute for physical read indices
e4c429f4
Final actual changes
6e68685a
Adress some todos
ed7d437c
Rename input_outputs
90043874
Modify the behavior of async
188ee868
Fix bugs
8f728f91
Added async tests
b33a6caa
Fix test
d48c2bd1
Remodel example
353c2619
Fix offload test
9b0dc901
Fix real cause of offload fail
cce892a0
Nits
6cf6b97f
Propagate use_async
f4a6a67d
Performance fixes 1
7ef9f541
More flexibility for cuda graphs
e47d0ea0
Remodeled the read and write indices
2c171fc5
Review compliance
0c70eb31
More doc and beautifull ascii
df57dae2
Style
66e05bc0
Fixes for end of generation
1b7e1d60
Review compliance
e538826f
Merge branch 'main' into cb-async-lfg
fe2596e9
More tokens to pass test
736370b8
Login to write a write a comment.
Login via GitHub