DeepSpeed
245c3a98 - Check AutoSP sequence length and head count divisibility (#8675)

Commit
51 days ago
Check AutoSP sequence length and head count divisibility (#8675) ## What breaks AutoSP shards the sequence across the SP ranks, and the Q/K/V all-to-all trades heads for sequence (`[B, N, S/P, H] -> [B, N/P, S, H]`). So the sequence length and the head counts must both be divisible by `sequence_parallel_size`. Neither is checked. Reproduced on 2 ranks (sp=2) with a small SDPA model compiled through `apply_autosp` + Inductor (the default AutoSP backend), torch 2.13: | case | master | |---|---| | 4 query heads, 1 KV head (GQA) | runs; **Q/K/V projection gradients are exactly 0** | | 3 query heads | `RuntimeError: shape '[2, (s38//2), 24]' is invalid for input of size 32*((s38//2))` at compile time | | sequence length 15 | `AssertionError: expected size 2==2, stride 224==240 at dim=0; ... expected size 14==15 ...` in `autosp::all_to_all` | | sequence length 15, eager backend | runs; the last token is silently dropped | The GQA case is the worst one. For KV heads, the all-to-all reshapes `N` heads into `N // P = 0` heads per rank. Under Inductor this yields an empty attention output and training continues without error, while attention does not learn: ``` rank0: OK loss=4.1853 k.grad.norm=0 v.grad.norm=0 q.grad.norm=0 ``` For the sequence length, every rank takes `S // P` tokens, so `S % P` tokens are dropped. The fake all-to-all restores the full symbolic `S` afterwards, which is where the Inductor stride assertion comes from. ## Fix - **Heads**: `pass_insert_attention_all_to_all` checks the query, key and value head counts of every SDPA call. They are checked separately because with GQA the K/V heads can be fewer than the query heads. The `autosp::all_to_all` op also checks the head count before the collective, since it can be called outside the pass. - **Sequence length**: the length is a dynamic dimension, so it can only be checked when the graph runs. `create_shard_offsets` inserts an `aten._assert_scalar` on `S % P == 0` next to the shard offsets. That is the op Dynamo itself emits for runtime asserts, and unlike `torch._assert` it is not stripped under `python -O`. - **Docs**: the AutoSP note in `training.rst` now states both requirements. With the fix, the same cases give: ``` ValueError: AutoSP requires the number of key heads (1) to be divisible by sequence_parallel_size (2) ValueError: AutoSP requires the number of query heads (3) to be divisible by sequence_parallel_size (2) RuntimeError: AutoSP requires the sequence length to be divisible by sequence_parallel_size (2) ``` Divisible lengths reuse the compiled graph as before. For example, after compiling with S=16, S=32 runs without a recompile, and S=15 and S=17 raise the error above. ## Tests `TestAutoSPDivisibilityValidation` in `tests/unit/v1/compile/test_compile_autosp.py` contains plain pytest tests that run IR-level passes with a mocked `sp_size`: | test | master | this PR | |---|---|---| | `test_shard_offsets_for_divisible_sequence_length[16, 32]` | PASSED | PASSED | | `test_shard_offsets_reject_non_divisible_sequence_length` | FAILED | PASSED | | `test_rejects_non_divisible_attention_heads[mha]` (3 query heads) | FAILED | PASSED | | `test_rejects_non_divisible_attention_heads[gqa]` (4 query heads, 1 KV head) | FAILED | PASSED | | `test_accepts_grouped_query_attention_with_divisible_heads` (4 query heads, 2 KV heads) | PASSED | PASSED | | `test_all_to_all_rejects_non_divisible_heads_before_the_collective` | FAILED | PASSED | The existing IR-level tests in the same file also pass (29 passed). Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Author
Parents
Loading