transformers
3117d533 - Continuous batching: find the sharded key/value projections under any decoder path (#50366)

Commit
1 day ago
Continuous batching: find the sharded key/value projections under any decoder path (#50366) * Continuous batching: find the sharded key/value projections under any decoder path A vision-language model's TP plan keys start with model.language_model., so the engine took its key/value heads as replicated and sized the cache with all of them on every rank: Qwen3.5 at tp=2 fails on the first decode step. * Nit --------- Co-authored-by: remi-or <remi.pierre_o@orange.fr>
Author
Parents
Loading