transformers
Run multi-head latent attention models under continuous batching
#50365

Open

Run multi-head latent attention models under continuous batching #50365

qgallouedec wants to merge 5 commits into cb-indexed-attention from cb-mla

Run multi-head latent attention models under continuous batching

666a54ed

Merge branch 'cb-cache-layout' into cb-mla

06e8101b

Merge branch 'cb-indexed-attention' into cb-mla

93ea3ae9

Cache the latent and the rotary key of latent attention as one row

f78dc44f

Say that latent attention caches one row in the architecture doc

457856ae

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone