DeepSpeed
729d5aeb - fix(rollout): forward top-k sampling configuration (#8676)

Commit
10 days ago
fix(rollout): forward top-k sampling configuration (#8676) ## Summary `SamplingConfig` exposes `top_k`, but `HybridEngineRollout` did not forward it to Transformers generation. As a result, sampling callers could not apply their configured top-k limit and the default non-positive setting could fall through to Transformers' own behavior. This PR: - forwards `top_k` when `HybridEngineRollout` uses random sampling; - treats `top_k <= 0` as disabled by passing Transformers its disabled value; - preserves greedy generation, CUDA graph capture, and continuous batching behavior; - removes the unused local top-p sampler, since rollout sampling delegates to Transformers. ## Validation - verifies top-k forwarding during random sampling; - verifies prompt expansion for multiple samples; - verifies EOS padding and the `eos_token_id=None` ZeRO lockstep contract; - verifies disabled `top_k` values (`-1` and `0`); - focused regression tests: 4 passed; - Changed-file pre-commit checks: passed Thank you for your review. Signed-off-by: nathon-lee <leejianwoo@gmail.com>
Author
Parents
Loading