transformers
Size the paged cache from the rows and states the layers declare
#50356
Open
Open
Size the paged cache from the rows and states the layers declare #50356
qgallouedec
wants to merge
5 commits into
cb-linear-attention
from
cb-cache-layout
Size the paged cache from the rows the layers declare, not from the m…
7d8ca02e
Declare cache rows and states by name, and keep every layer's states …
b07b9f91
Merge branch 'cb-linear-attention' into cb-cache-layout
1aa1ba90
Collapse the masks to one tensor only for models with a single layer …
c548a7b4
qgallouedec
changed the title
Skip the fill pass over the states when the fill is zero
38935dcb
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub