DeepSpeed
663de973 - [Ulysses] Add opt-in uneven-shard rejection and FPDT padding (#9059)

Commit
5 days ago
[Ulysses] Add opt-in uneven-shard rejection and FPDT padding (#9059) ## Problem [`DistributedAttention`](https://github.com/jschneider-canva/DeepSpeed/blob/5a8b4a67b4a93aeec42a1b403da18647a7b04ebe/deepspeed/sequence/layer.py) assumes equal sequence lengths across sequence-parallel ranks. Unequal shards can hang, abort, or silently corrupt results, depending on backend and version ([#7384](https://github.com/deepspeedai/DeepSpeed/issues/7384), [#8351](https://github.com/deepspeedai/DeepSpeed/pull/8351)). A two-GPU reproduction hangs without a mask or fails with a callback shape error when masked. Separately, [`FPDT_InputConstruct`](https://github.com/jschneider-canva/DeepSpeed/blob/5a8b4a67b4a93aeec42a1b403da18647a7b04ebe/deepspeed/sequence/fpdt_layer.py) rejects sequence lengths that do not fill its global chunks, requiring callers to pad manually. [#7868](https://github.com/deepspeedai/DeepSpeed/pull/7868) proposed zero-padding Q/K/V inside attention. In a five-token reproduction, that produced maximum absolute errors of 0.24 in outputs and 0.32 in Q gradients against an unpadded reference, and unmasked zero keys still receive softmax probability. ## Approach Report unequal shards as an error before generic attention data exchange and pad FPDT inputs under its causal contract. - **Ulysses:** add local sequence-scatter divisibility assertions and opt-in `DistributedAttention(..., validate_shards=True)`. The enabled check gathers Q/K/V sequence lengths across the SP group and raises `ValueError` on every participating rank before attention data exchange. Q and K/V may differ from each other; each must agree across ranks. Validation defaults off. - **FPDT:** validate sequence/chunk/SP geometry and rank, then pad incomplete sequences to the global chunk size. Append zero inputs, label `0`, zero loss weights, and continued position IDs. Padding requires `loss_mask` and `attention_mask=None`. The padding and normalization contract is documented in the FPDT tutorial and constructor docstring. - Export `DistributedAttention` from [`deepspeed.sequence`](https://github.com/jschneider-canva/DeepSpeed/blob/5a8b4a67b4a93aeec42a1b403da18647a7b04ebe/deepspeed/sequence/__init__.py); keep `_SeqAllToAll` private. Outputs and gradients were verified against an unpadded reference within numerical tolerances. Enable validation consistently across all SP ranks - this adds one metadata all-gather and host synchronization per forward. Generic-attention callers still own the padding and masks. For FPDT, apply the returned loss mask and normalize by the SP-global sum of loss weights. <details> <summary>PKUWZP’s review items on #7868</summary> | # | Item | Resolution here | |---|------|-----------------| | 1 | Padding changes attention normalization | No internal generic-attention padding; FPDT uses causal tail padding. | | 2 | Per-forward collective overhead | No new default collective; validation adds one metadata all-gather when enabled. | | 3 | Backward correctness | Output, loss, input/QKV/head gradients and padding invariance tested. | | 4 | Attention masks | Reject non-`None` masks when padding is required; preserve aligned pass-through. | | 5 | Chunk divisibility | Pad to global chunk size; validate divisibility by SP size. | | 6 | Original-length field and padded loss | No unused field; require loss masking and zero padded loss weights. | | 7 | Unused imports | No extra attention/max-length imports; F401 checks passed. | | 8 | Private export | Export `DistributedAttention` only. | | 9 | Regression coverage | Add uneven rejection, valid/caller-padded attention, and CPU/GPU FPDT tests; [#8317](https://github.com/deepspeedai/DeepSpeed/pull/8317) already covers the shape fix. | </details> ## Testing **137 passed, zero skips** across the [FPDT constructor](https://github.com/jschneider-canva/DeepSpeed/blob/5a8b4a67b4a93aeec42a1b403da18647a7b04ebe/tests/unit/v1/sequence_parallelism/test_fpdt_inputs.py), [Ulysses](https://github.com/jschneider-canva/DeepSpeed/blob/5a8b4a67b4a93aeec42a1b403da18647a7b04ebe/tests/unit/v1/sequence_parallelism/test_ulysses.py), and [ALST](https://github.com/jschneider-canva/DeepSpeed/blob/5a8b4a67b4a93aeec42a1b403da18647a7b04ebe/tests/unit/ulysses_alst/test_ulysses_sp_hf.py) files on four H200s. The six affected rejection cases passed again on two H200s after test deduplication. Coverage includes: - Unequal shards in both rank orders, both tensor layouts, valid equal shards, cross-attention, and independent SP subgroups. - **39 CPU constructor tests:** geometry/errors, padding values, positions, ordering, reconstruction, optional fields, and input immutability. - **FPDT at SP2:** aligned length 24, odd length 19, and short length 3, with both offload requests. Comparisons exercise actual attention and loss backward, globally normalized loss, input/QKV/head gradients, caller padding, and padding perturbations. - Length 19 exercises both multi-chunk attention paths. Length 3 uses the existing one-chunk non-offloading fallback, including when offloading is requested. - Changed-file pre-commit and strict FPDT F401 checks passed. Environment: Python 3.11.10, Torch 2.11.0+cu128, CUDA 12.8, FA2 2.8.3, pytest 8.3.5, NVIDIA driver 590.48.01. New numerical coverage uses FP32 SDPA for Ulysses and FP16/SP2/zero dropout for FPDT. <details> <summary>Test commands</summary> ```bash # CPU environment without FA2 CUDA_VISIBLE_DEVICES='' DS_ACCELERATOR=cpu PYTHONPATH="$PWD" \ python -m pytest -q tests/unit/v1/sequence_parallelism/test_fpdt_inputs.py # Full suite: four H200s CUDA_VISIBLE_DEVICES=0,1,2,3 PYTHONPATH="$PWD" \ HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \ python -m pytest --forked -q -rs \ tests/unit/v1/sequence_parallelism/test_fpdt_inputs.py \ tests/unit/v1/sequence_parallelism/test_ulysses.py \ tests/unit/ulysses_alst/test_ulysses_sp_hf.py --maxfail=1 ``` </details> The pre-existing `softmax().log()` instability is being addressed separately in [#9068](https://github.com/deepspeedai/DeepSpeed/pull/9068). This PR documents zeroing losses at zero-weight positions before weighting, avoiding `inf * 0 = NaN` for masked tokens. ## Duplicate Work Addresses [#7384](https://github.com/deepspeedai/DeepSpeed/issues/7384) and picks up the remaining work from [#7868](https://github.com/deepspeedai/DeepSpeed/pull/7868). [#8351](https://github.com/deepspeedai/DeepSpeed/pull/8351) proposed the local divisibility assertion, this PR adds cross-rank validation and FPDT padding. The sibling ALST data-loader adapter already rejects non-divisible lengths and is unchanged. Our pre-submission search found no other open implementation of this approach. Builds on [@harshang03](https://github.com/harshang03)’s proposal and [@PKUWZP](https://github.com/PKUWZP)’s review of [#7868](https://github.com/deepspeedai/DeepSpeed/pull/7868). AI assistance was used for implementation, testing, review, and preparation of this PR. --------- Signed-off-by: Jacob Schneider-Martin <jschneider@canva.com>
Parents
Loading