Check AutoSP sequence length and head count divisibility (#8675)
## What breaks
AutoSP shards the sequence across the SP ranks, and the Q/K/V all-to-all
trades heads for sequence (`[B, N, S/P, H] -> [B, N/P, S, H]`). So the
sequence length and the head counts must both be divisible by
`sequence_parallel_size`. Neither is checked. Reproduced on 2 ranks
(sp=2) with a small SDPA model compiled through `apply_autosp` +
Inductor (the default AutoSP backend), torch 2.13:
| case | master |
|---|---|
| 4 query heads, 1 KV head (GQA) | runs; **Q/K/V projection gradients
are exactly 0** |
| 3 query heads | `RuntimeError: shape '[2, (s38//2), 24]' is invalid
for input of size 32*((s38//2))` at compile time |
| sequence length 15 | `AssertionError: expected size 2==2, stride
224==240 at dim=0; ... expected size 14==15 ...` in `autosp::all_to_all`
|
| sequence length 15, eager backend | runs; the last token is silently
dropped |
The GQA case is the worst one. For KV heads, the all-to-all reshapes `N`
heads into `N // P = 0` heads per rank. Under Inductor this yields an
empty attention output and training continues without error, while
attention does not learn:
```
rank0: OK loss=4.1853 k.grad.norm=0 v.grad.norm=0 q.grad.norm=0
```
For the sequence length, every rank takes `S // P` tokens, so `S % P`
tokens are dropped. The fake all-to-all restores the full symbolic `S`
afterwards, which is where the Inductor stride assertion comes from.
## Fix
- **Heads**: `pass_insert_attention_all_to_all` checks the query, key
and value head counts of every SDPA call. They are checked separately
because with GQA the K/V heads can be fewer than the query heads. The
`autosp::all_to_all` op also checks the head count before the
collective, since it can be called outside the pass.
- **Sequence length**: the length is a dynamic dimension, so it can only
be checked when the graph runs. `create_shard_offsets` inserts an
`aten._assert_scalar` on `S % P == 0` next to the shard offsets. That is
the op Dynamo itself emits for runtime asserts, and unlike
`torch._assert` it is not stripped under `python -O`.
- **Docs**: the AutoSP note in `training.rst` now states both
requirements.
With the fix, the same cases give:
```
ValueError: AutoSP requires the number of key heads (1) to be divisible by sequence_parallel_size (2)
ValueError: AutoSP requires the number of query heads (3) to be divisible by sequence_parallel_size (2)
RuntimeError: AutoSP requires the sequence length to be divisible by sequence_parallel_size (2)
```
Divisible lengths reuse the compiled graph as before. For example, after
compiling with S=16, S=32 runs without a recompile, and S=15 and S=17
raise the error above.
## Tests
`TestAutoSPDivisibilityValidation` in
`tests/unit/v1/compile/test_compile_autosp.py` contains plain pytest
tests that run IR-level passes with a mocked `sp_size`:
| test | master | this PR |
|---|---|---|
| `test_shard_offsets_for_divisible_sequence_length[16, 32]` | PASSED |
PASSED |
| `test_shard_offsets_reject_non_divisible_sequence_length` | FAILED |
PASSED |
| `test_rejects_non_divisible_attention_heads[mha]` (3 query heads) |
FAILED | PASSED |
| `test_rejects_non_divisible_attention_heads[gqa]` (4 query heads, 1 KV
head) | FAILED | PASSED |
| `test_accepts_grouped_query_attention_with_divisible_heads` (4 query
heads, 2 KV heads) | PASSED | PASSED |
| `test_all_to_all_rejects_non_divisible_heads_before_the_collective` |
FAILED | PASSED |
The existing IR-level tests in the same file also pass (29 passed).
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>