DeepSpeed
8b7422e2 - fix(rollout): pass a pre-built 4-D mask to the static graph decode forward (#9004)

Commit
10 days ago
fix(rollout): pass a pre-built 4-D mask to the static graph decode forward (#9004) Closes #9000. ## Problem With `use_graph_capture=True` and no `continuous_batch_size`, `_generate_graph` gives the captured decode forward a 2-D attention mask, so Transformers builds the causal mask during capture. That fails on the supported versions: - **4.57.x with SDPA:** `padding_mask.all()` runs on the host and breaks capture with "operation not permitted when stream is capturing" (reported by @0z5a on #8197). - **5.x:** Transformers calls `get_query_offset`, which `DeepSpeedStaticCache` doesn't have. ## Fix The captured forward now gets a pre-built 4-D additive mask. The mask is refreshed before each replay, and prefill is unchanged. The fix is cherry-picked from @0z5a's commit. `_generate_graph` also raises a `ValueError` for `flash_attention_2`, as the continuous graph path does, because it can't use a 4-D mask. #8602 includes the same mask change, but it's a much larger PR that will take a while to merge. #8852 is still needed for continuous batching without graphs on 5.x. ## Test results Run on 1 NVIDIA A40 with torch 2.14.1+cu130, using real Hugging Face models without a DeepSpeed engine. ```bash python -m pytest tests/unit/v1/test_hybrid_engine_rollout_v1.py tests/unit/runtime/rollout/ tests/unit/utils/test_static_cache.py ``` | Transformers | New tests with this PR | Rollout and cache suites with this PR | | --- | --- | --- | --- | | 4.57.6 | 4 passed | 89 passed, 2 skipped | | 5.19.0 | 4 passed | 89 passed, 2 skipped | | 5.22.1 | 4 passed | 89 passed, 2 skipped | ## Compatibility - **Validated:** CUDA on an A40; fp32, fp16 and bf16; eager and SDPA attention; Transformers 4.57.6, 5.19.0 and 5.22.1. - **Not validated:** - accelerators other than CUDA - hybrid models - the `flash_attention_2` rejection on a real FlashAttention model --------- Signed-off-by: Puyuan Yao <williamyao034@gmail.com> Co-authored-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Author
Parents
Loading