DeepSpeed
388f6a3f - Fix static rollout graph decoding regressions (#9127)

Commit
21 hours ago
Fix static rollout graph decoding regressions (#9127) ## Summary Fixes #9126. - Size the static rollout KV cache from the actual local prefill tensors instead of guessed global/TP head counts. - Pass an additive 4-D mask to eager/SDPA attention while retaining a separate boolean mask for the captured step kernel. - Apply repetition penalties on the static graph path, reset capture-time mask mutations before replay, and restrict the full-step kernel to the single-row, unpenalized greedy fast path. ## Root cause The segment-KI rollout change eagerly initialized `DeepSpeedStaticCache` from global configuration head counts. That allocation can disagree with AutoTP's local KV heads, causing either a copy shape failure or silent wrong-token generation. The same change also passed a boolean 4-D mask directly to attention implementations that expect additive masks, while graph/Python token selection bypassed the repetition penalty. Together these account for the nightly static-graph failures in `tests/unit/v1/test_hybrid_engine_rollout_v1.py`. The batched graph step did not need the full-step argmax kernel because Python already selects each row's token. The kernel is now explicitly limited to batch size one; the b=1 EOS check remains amortized over 16 replays because at most 15 trailing steps are padded away, avoiding a synchronization on every replay. Batched decoding continues checking after every step because its dead work is multiplied by the number of active rows. ## Duplicate-work check #9077 is primarily a static graph-reuse performance PR, not the dedicated #9126 fix, but its rebase onto #8602 independently absorbed the same local-KV, additive-mask, and repetition-penalty corrections while preserving that performance work. #9127 remains the isolated hotfix for the nightly regression. If #9127 is merged first, #9077 should rebase onto it and drop the duplicated correctness hunks; its remaining scope is graph reuse and lifecycle. #9079 is stacked on #9077 and separately handles whole-step graph capture, including replacement of the Python token-history loop. ## Validation Hardware: 1x Intel Arc Pro B70 (16 devices visible), Level Zero XPU; driver version is not exposed by the environment. Software: torch `2.14.0.dev20260804+xpu`, Transformers `5.17.0`. ```bash python -m pytest -q -p no:cacheprovider -o addopts= \ tests/unit/utils/test_static_cache.py \ 'tests/unit/v1/test_hybrid_engine_rollout_v1.py::test_static_graph_attention_mask_uses_additive_values' \ 'tests/unit/v1/test_hybrid_engine_rollout_v1.py::test_static_graph_uses_prefill_kv_shape' \ 'tests/unit/v1/test_hybrid_engine_rollout_v1.py::test_static_graph_full_step_kernel_preconditions' \ 'tests/unit/v1/test_hybrid_engine_rollout_v1.py::test_static_graph_matches_native_llama_generation[eager]' # 26 passed in the combined focused set pre-commit run --files \ deepspeed/runtime/rollout/hybrid_engine_rollout.py \ tests/unit/v1/test_hybrid_engine_rollout_v1.py # all hooks passed ``` The broad two-file run produced 34 passes and 17 skips before the added direct regressions; its remaining failures were unsupported Level Zero SDPA/distributed operations (`UR_RESULT_ERROR_UNSUPPORTED_FEATURE`), not assertion regressions in the changed path. The new direct regressions assert that visible positions are additive zero and masked positions are `finfo.min`, and that the full-step kernel is not mounted for repetition penalties or batched decode. Existing end-to-end tests compare generated tokens against Hugging Face and cover the penalty result on CUDA CI. ## Compatibility coverage - Validated: static graph replay on Intel XPU; AutoTP-like local KV head counts 4/2/1; eager Llama attention; greedy decoding; additive mask values; full-step-kernel preconditions; cache replay/position/compaction contracts. - Not validated locally: NVIDIA CUDA, SDPA on CUDA, ROCm, sampled/repetition-penalty end-to-end rollout on a CUDA device, Hybrid/linear-attention models, and multi-rank AutoTP. These require hardware/backend coverage outside this environment. - ZeRO stage/offload are not affected. No public API, config, checkpoint, convergence, throughput, or peak-memory behavior is changed. The per-step `torch.cat` used by repetition penalty remains O(n²), matching the existing continuous rollout implementation. Optimizing only this static path is small (roughly 30–60 minutes plus a focused test), but unifying it with variable-length continuous rollout would take roughly 2–4 hours and require broader compaction/padding/CUDA validation. It is intentionally deferred from this correctness fix and can be handled as a small follow-up performance PR, primarily benefiting `repetition_penalty != 1`; the default penalty path is unaffected. AI assistance was used for investigation, implementation, and test preparation; the submitting human reviewed the final diff and is responsible for the DCO sign-off. --------- Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com> Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>
Author
Parents
Loading