Continuous batching refactor #40426
ArthurZucker
merged
27 commits into
huggingface:main
from
remi-or:conbat
ArthurZucker
approved these changes
on 2025-08-26
Rework of the CB example
9495e0e1
Further rework of CB example
cce99f07
Refactor PA cache, slice on tokens, add debug prints -- WIP
74a0d73d
Slice cache -- WIP
79118c53
Added a mechanism to check batched outputs in CB script
dc53ad60
Less logging, debug flag for slice, !better reset! -- WIP
b107785c
QOL and safety margins
bababa46
Refactor and style
f01e9db4
Better saving of cb example
3cffe20e
Fix
7cd70ac1
Fixes and QOL
2933099b
Mor einformations about metrics
bfcf6117
Further logging
f000b17d
Style
042e87dd
Licenses
604fe6e5
Removed some comments
ef635477
Add a slice input flag
d403b02f
Fix in example
023774fd
Added back some open-telemetry deps
c327f08d
Removed some aux function
173b497a
Added FA2 option to example script
fff2ee8a
Fixed math (all of it)
7353aef1
Added a simple example
0de06e30
Renamed core to classes
8325b376
Made allocation of attention mask optionnal
7dee44e4
Style
3f17daf8
Merge branch 'main' into conbat
463ea913
Login to write a write a comment.
Login via GitHub