DeepSpeed
8b05554f - perf(rollout): amortize the b>1 EOS check like the b=1 path

Commit
4 days ago
perf(rollout): amortize the b>1 EOS check like the b=1 path eos_mask.all() is a device->host sync on every decode step; it serializes the ~50us of Python glue against the replay, and also runs when eos_token_id is None where the mask can never become true. Check every eos_check_every=16 steps instead, mirroring the b=1 path. Rows that hit EOS keep accumulating in eos_mask, so the break lands on the first check boundary after the last row's EOS, and tokens generated past a row's EOS are padded by _pad_after_eos -- the output is unchanged. Suggested by @jinyouzhi. Measured on L20 (2x for TP=2), Qwen3.5-4B, b=32 greedy decode, interleaved A/B with medians: TP=2 1988 -> 1993 tok/s (+0.3%, non-overlapping ranges); TP=1 within noise (+0.1%). The gain scales with the ratio of the Python glue to the step time, so it matters most at small batch / faster GPUs where you profiled it. Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
Author
Parents
Loading