transformers
d85573b1 - Fix mask creation not being skipped under `torch.compile` (#48975)

Commit
53 days ago
Fix mask creation not being skipped under `torch.compile` (#48975) * Skip mask creation under torch.compile when there is no padding mask `_ignore_causal_mask_sdpa` and `_ignore_bidirectional_mask_sdpa` bailed out on `is_tracing(padding_mask)`, which is always True under `torch.compile`. As a result the 4D mask was materialized even without a padding mask, so sdpa could never dispatch to its flash/oneDNN kernels via `is_causal`. Whether a padding mask is passed at all is a static property that dynamo guards on. Only the checks reading the mask values are a data-dependent control flow, so the tracing guard is moved down to those, and `torch.export` keeps bailing out early through `is_torchdynamo_exporting()`. OPT additionally overwrote `attention_mask` with a dense all-ones mask to infer its learned positional embeddings, which then leaked into `create_causal_mask` and defeated the skip. The dense mask is now kept in a local variable. Fixes #48924 Fixes #48925 * Gate the compile skip test on torch>=2.14 and return bools from the skip helpers `is_torchdynamo_exporting()` maps to `torch.compiler.is_exporting()`, which dynamo hard-coded to `True` in its `tracing_state_functions` table, i.e. it also reported exporting under plain `torch.compile`. That was only fixed by pytorch#176499, released in torch 2.14, so on older versions the export guard still triggers under compile and the mask keeps being materialized (conservative and correct, just without the speedup). Since setup.py allows torch>=2.5, gate the new compile test with `@require_torch_greater_or_equal("2.14")` so it does not fail on older versions, and document the dependency next to the guard. Note that this cannot be worked around from user code: on 2.7.1 every tracing-state signal has identical values under `torch.compile` and strict `torch.export`, and reading `torch.compiler._is_exporting_flag` directly is unsafe because it is `False` during strict export and would bake the skip into the exported program. Also return a Python `bool` via `bool(fast_all(padding_mask))` from `_ignore_bidirectional_mask_sdpa` / `_can_skip_bidirectional_mask_xpu` instead of a 0-dim tensor from `padding_mask.all()`, matching their `-> bool` annotation and the `fast_all` helper already used by the causal path. * Reduce the diff to a single shared tracing guard Address review feedback: instead of reordering the skip helpers, keep their original structure and only replace the tracing guard. The new helper makes the export check explicit and removes the duplicated `is_tracing` calls. Verified to be decision-identical to the previous version across the full grid of padding masks, query lengths, cache offsets and local attention sizes, in eager, `torch.compile` and `torch.export`. * Move the OPT fix to a separate PR The `modeling_opt.py` change is independent of the mask skipping logic and is better reviewed on its own, so this PR is now limited to `masking_utils.py`. * Simplify the tracing guard docstring * Inline the tracing guard and keep the original comments Drops the helper introduced earlier: the condition is now inlined at the three call sites, so masking_utils.py is down to +7/-3 with every original comment restored. Also rephrases the torch 2.14 note, which was misleading: dynamo replaces calls to torch.compiler.is_exporting() by a constant True while tracing, it is not the function itself that is hard-coded. * Address review nits: single NOTE block, comment in docstring, no loop
Author
Parents
Loading