Continuous batching for linear attention layers #50350
qgallouedec
wants to merge
7 commits into
main
from
cb-linear-attention
Continuous batching for linear attention layers
2b96d3db
Wait for each warmup forward before the next one rewrites the static …
64def275
Let the linear attention layers declare the states the paged cache ke…
ab7c8fe5
Leave the warmup stream sync to its own pull request
b17183b5
Type the allocator as the linear one where the cache reads its state …
237b61ba
Name the declaration cache_states: the states a layer asks the cache …
e05b2713
Keep the engine's masks as a dict on hybrid models
2d67d9b7
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub