fix(rollout): pass a pre-built 4-D mask to the static graph decode forward (#9004)
Closes #9000.
## Problem
With `use_graph_capture=True` and no `continuous_batch_size`,
`_generate_graph` gives the captured decode forward a 2-D attention
mask, so Transformers builds the causal mask during capture. That fails
on the supported versions:
- **4.57.x with SDPA:** `padding_mask.all()` runs on the host and breaks
capture with "operation not permitted when stream is capturing"
(reported by @0z5a on #8197).
- **5.x:** Transformers calls `get_query_offset`, which
`DeepSpeedStaticCache` doesn't have.
## Fix
The captured forward now gets a pre-built 4-D additive mask. The mask is
refreshed before each replay, and prefill is unchanged. The fix is
cherry-picked from @0z5a's commit.
`_generate_graph` also raises a `ValueError` for `flash_attention_2`, as
the continuous graph path does, because it can't use a 4-D mask.
#8602 includes the same mask change, but it's a much larger PR that will
take a while to merge. #8852 is still needed for continuous batching
without graphs on 5.x.
## Test results
Run on 1 NVIDIA A40 with torch 2.14.1+cu130, using real Hugging Face
models without a DeepSpeed engine.
```bash
python -m pytest tests/unit/v1/test_hybrid_engine_rollout_v1.py tests/unit/runtime/rollout/ tests/unit/utils/test_static_cache.py
```
| Transformers | New tests with this PR | Rollout and cache suites with
this PR |
| --- | --- | --- | --- |
| 4.57.6 | 4 passed | 89 passed, 2 skipped |
| 5.19.0 | 4 passed | 89 passed, 2 skipped |
| 5.22.1 | 4 passed | 89 passed, 2 skipped |
## Compatibility
- **Validated:** CUDA on an A40; fp32, fp16 and bf16; eager and SDPA
attention; Transformers 4.57.6, 5.19.0 and 5.22.1.
- **Not validated:**
- accelerators other than CUDA
- hybrid models
- the `flash_attention_2` rejection on a real FlashAttention model
---------
Signed-off-by: Puyuan Yao <williamyao034@gmail.com>
Co-authored-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>