Fix static rollout graph decoding regressions (#9127)
## Summary
Fixes #9126.
- Size the static rollout KV cache from the actual local prefill tensors
instead of guessed global/TP head counts.
- Pass an additive 4-D mask to eager/SDPA attention while retaining a
separate boolean mask for the captured step kernel.
- Apply repetition penalties on the static graph path, reset
capture-time mask mutations before replay, and restrict the full-step
kernel to the single-row, unpenalized greedy fast path.
## Root cause
The segment-KI rollout change eagerly initialized `DeepSpeedStaticCache`
from global configuration head counts. That allocation can disagree with
AutoTP's local KV heads, causing either a copy shape failure or silent
wrong-token generation. The same change also passed a boolean 4-D mask
directly to attention implementations that expect additive masks, while
graph/Python token selection bypassed the repetition penalty. Together
these account for the nightly static-graph failures in
`tests/unit/v1/test_hybrid_engine_rollout_v1.py`.
The batched graph step did not need the full-step argmax kernel because
Python already selects each row's token. The kernel is now explicitly
limited to batch size one; the b=1 EOS check remains amortized over 16
replays because at most 15 trailing steps are padded away, avoiding a
synchronization on every replay. Batched decoding continues checking
after every step because its dead work is multiplied by the number of
active rows.
## Duplicate-work check
#9077 is primarily a static graph-reuse performance PR, not the
dedicated #9126 fix, but its rebase onto #8602 independently absorbed
the same local-KV, additive-mask, and repetition-penalty corrections
while preserving that performance work. #9127 remains the isolated
hotfix for the nightly regression. If #9127 is merged first, #9077
should rebase onto it and drop the duplicated correctness hunks; its
remaining scope is graph reuse and lifecycle. #9079 is stacked on #9077
and separately handles whole-step graph capture, including replacement
of the Python token-history loop.
## Validation
Hardware: 1x Intel Arc Pro B70 (16 devices visible), Level Zero XPU;
driver version is not exposed by the environment. Software: torch
`2.14.0.dev20260804+xpu`, Transformers `5.17.0`.
```bash
python -m pytest -q -p no:cacheprovider -o addopts= \
tests/unit/utils/test_static_cache.py \
'tests/unit/v1/test_hybrid_engine_rollout_v1.py::test_static_graph_attention_mask_uses_additive_values' \
'tests/unit/v1/test_hybrid_engine_rollout_v1.py::test_static_graph_uses_prefill_kv_shape' \
'tests/unit/v1/test_hybrid_engine_rollout_v1.py::test_static_graph_full_step_kernel_preconditions' \
'tests/unit/v1/test_hybrid_engine_rollout_v1.py::test_static_graph_matches_native_llama_generation[eager]'
# 26 passed in the combined focused set
pre-commit run --files \
deepspeed/runtime/rollout/hybrid_engine_rollout.py \
tests/unit/v1/test_hybrid_engine_rollout_v1.py
# all hooks passed
```
The broad two-file run produced 34 passes and 17 skips before the added
direct regressions; its remaining failures were unsupported Level Zero
SDPA/distributed operations (`UR_RESULT_ERROR_UNSUPPORTED_FEATURE`), not
assertion regressions in the changed path.
The new direct regressions assert that visible positions are additive
zero and masked positions are `finfo.min`, and that the full-step kernel
is not mounted for repetition penalties or batched decode. Existing
end-to-end tests compare generated tokens against Hugging Face and cover
the penalty result on CUDA CI.
## Compatibility coverage
- Validated: static graph replay on Intel XPU; AutoTP-like local KV head
counts 4/2/1; eager Llama attention; greedy decoding; additive mask
values; full-step-kernel preconditions; cache replay/position/compaction
contracts.
- Not validated locally: NVIDIA CUDA, SDPA on CUDA, ROCm,
sampled/repetition-penalty end-to-end rollout on a CUDA device,
Hybrid/linear-attention models, and multi-rank AutoTP. These require
hardware/backend coverage outside this environment.
- ZeRO stage/offload are not affected. No public API, config,
checkpoint, convergence, throughput, or peak-memory behavior is changed.
The per-step `torch.cat` used by repetition penalty remains O(n²),
matching the existing continuous rollout implementation. Optimizing only
this static path is small (roughly 30–60 minutes plus a focused test),
but unifying it with variable-length continuous rollout would take
roughly 2–4 hours and require broader compaction/padding/CUDA
validation. It is intentionally deferred from this correctness fix and
can be handled as a small follow-up performance PR, primarily benefiting
`repetition_penalty != 1`; the default penalty path is unaffected.
AI assistance was used for investigation, implementation, and test
preparation; the submitting human reviewed the final diff and is
responsible for the DCO sign-off.
---------
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>