Fix ZeRO-3 synchronization during OPSD rollout (#8264)
## Summary
- disable rank-local EOS early exit in the OPSD Hybrid Engine Hugging
Face rollout path
- keep ZeRO-3 decode forwards and parameter collectives in the same
order on every data-parallel rank
- add a CPU unit test that verifies the collective-safe generation
argument
Fixes #8262.
## Why
ZeRO-3 gathers partitioned parameters during every autoregressive decode
forward. If one data-parallel rank emits EOS and returns from
`generate()` while another rank continues decoding, the continuing rank
enters another parameter all-gather while the finished rank moves to a
later collective. The job then deadlocks.
Passing `eos_token_id=None` makes every rank execute exactly
`max_new_tokens` iterations. Output masking remains unchanged, so EOS
and padding tokens are still excluded from downstream response loss as
applicable.
## Validation
- `pytest -q tests/unit/runtime/rollout/test_hybrid_engine_rollout.py`
(`10 passed`)
- `pre-commit run --files
deepspeed/runtime/rollout/hybrid_engine_rollout.py
tests/unit/runtime/rollout/test_hybrid_engine_rollout.py`
- 2-step distributed reproducer completed on 8 x H200 and 8 x MI250
after the fix
- full prompt epoch and a separate 200-step OPSD run completed on 8 x
MI250
---------
Signed-off-by: LiRunGuo <li19107254665@gmail.com>
Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>
