transformers

666a54ed - Run multi-head latent attention models under continuous batching

Run multi-head latent attention models under continuous batching The latent attention layers (DeepSeek V2/V3, Kimi K2 and the models built on them) declare their compressed latent and rotary key as cache rows, write and read them through the cache, and rebuild the keys and values of all attended tokens before the attention. The cache hands keys and values through for layers declaring no key row, and only layers caching exactly a key and a value of the same size use the block table. The sparse attention indexers (DeepSeek V3.2, GLM-5, Mistral Large 4 and the models built on them) cache their keys as one more row and select on the packed batch under the engine's mask.