transformers
237b61ba - Type the allocator as the linear one where the cache reads its state views

Commit

2 days ago

Type the allocator as the linear one where the cache reads its state views ty refused state_views on a CacheAllocator; layer_to_allocator maps the linear layers to the linear allocator, which is the one that has it.

References

#50350 - Continuous batching for linear attention layers

#50356 - Size the paged cache from the rows and states the layers declare

#50359 - Run Qwen4-Exp under continuous batching

#50365 - Run multi-head latent attention models under continuous batching

#50369 - Decode multi-head latent attention from the cache pages with FlashMLA

Author

qgallouedec

Parents

b17183b5

transformers 237b61ba - Type the allocator as the linear one where the cache reads its state views

transformers
237b61ba - Type the allocator as the linear one where the cache reads its state views