[Ulysses] Add opt-in uneven-shard rejection and FPDT padding (#9059)
## Problem
[`DistributedAttention`](https://github.com/jschneider-canva/DeepSpeed/blob/5a8b4a67b4a93aeec42a1b403da18647a7b04ebe/deepspeed/sequence/layer.py)
assumes equal sequence lengths across sequence-parallel ranks. Unequal
shards can hang, abort, or silently corrupt results, depending on
backend and version
([#7384](https://github.com/deepspeedai/DeepSpeed/issues/7384),
[#8351](https://github.com/deepspeedai/DeepSpeed/pull/8351)). A two-GPU
reproduction hangs without a mask or fails with a callback shape error
when masked.
Separately,
[`FPDT_InputConstruct`](https://github.com/jschneider-canva/DeepSpeed/blob/5a8b4a67b4a93aeec42a1b403da18647a7b04ebe/deepspeed/sequence/fpdt_layer.py)
rejects sequence lengths that do not fill its global chunks, requiring
callers to pad manually.
[#7868](https://github.com/deepspeedai/DeepSpeed/pull/7868) proposed
zero-padding Q/K/V inside attention. In a five-token reproduction, that
produced maximum absolute errors of 0.24 in outputs and 0.32 in Q
gradients against an unpadded reference, and unmasked zero keys still
receive softmax probability.
## Approach
Report unequal shards as an error before generic attention data exchange
and pad FPDT inputs under its causal contract.
- **Ulysses:** add local sequence-scatter divisibility assertions and
opt-in `DistributedAttention(..., validate_shards=True)`. The enabled
check gathers Q/K/V sequence lengths across the SP group and raises
`ValueError` on every participating rank before attention data exchange.
Q and K/V may differ from each other; each must agree across ranks.
Validation defaults off.
- **FPDT:** validate sequence/chunk/SP geometry and rank, then pad
incomplete sequences to the global chunk size. Append zero inputs, label
`0`, zero loss weights, and continued position IDs. Padding requires
`loss_mask` and `attention_mask=None`. The padding and normalization
contract is documented in the FPDT tutorial and constructor docstring.
- Export `DistributedAttention` from
[`deepspeed.sequence`](https://github.com/jschneider-canva/DeepSpeed/blob/5a8b4a67b4a93aeec42a1b403da18647a7b04ebe/deepspeed/sequence/__init__.py);
keep `_SeqAllToAll` private.
Outputs and gradients were verified against an unpadded reference within
numerical tolerances.
Enable validation consistently across all SP ranks - this adds one
metadata all-gather and host synchronization per forward.
Generic-attention callers still own the padding and masks.
For FPDT, apply the returned loss mask and normalize by the SP-global
sum of loss weights.
<details>
<summary>PKUWZP’s review items on #7868</summary>
| # | Item | Resolution here |
|---|------|-----------------|
| 1 | Padding changes attention normalization | No internal
generic-attention padding; FPDT uses causal tail padding. |
| 2 | Per-forward collective overhead | No new default collective;
validation adds one metadata all-gather when enabled. |
| 3 | Backward correctness | Output, loss, input/QKV/head gradients and
padding invariance tested. |
| 4 | Attention masks | Reject non-`None` masks when padding is
required; preserve aligned pass-through. |
| 5 | Chunk divisibility | Pad to global chunk size; validate
divisibility by SP size. |
| 6 | Original-length field and padded loss | No unused field; require
loss masking and zero padded loss weights. |
| 7 | Unused imports | No extra attention/max-length imports; F401
checks passed. |
| 8 | Private export | Export `DistributedAttention` only. |
| 9 | Regression coverage | Add uneven rejection, valid/caller-padded
attention, and CPU/GPU FPDT tests;
[#8317](https://github.com/deepspeedai/DeepSpeed/pull/8317) already
covers the shape fix. |
</details>
## Testing
**137 passed, zero skips** across the [FPDT
constructor](https://github.com/jschneider-canva/DeepSpeed/blob/5a8b4a67b4a93aeec42a1b403da18647a7b04ebe/tests/unit/v1/sequence_parallelism/test_fpdt_inputs.py),
[Ulysses](https://github.com/jschneider-canva/DeepSpeed/blob/5a8b4a67b4a93aeec42a1b403da18647a7b04ebe/tests/unit/v1/sequence_parallelism/test_ulysses.py),
and
[ALST](https://github.com/jschneider-canva/DeepSpeed/blob/5a8b4a67b4a93aeec42a1b403da18647a7b04ebe/tests/unit/ulysses_alst/test_ulysses_sp_hf.py)
files on four H200s. The six affected rejection cases passed again on
two H200s after test deduplication.
Coverage includes:
- Unequal shards in both rank orders, both tensor layouts, valid equal
shards, cross-attention, and independent SP subgroups.
- **39 CPU constructor tests:** geometry/errors, padding values,
positions, ordering, reconstruction, optional fields, and input
immutability.
- **FPDT at SP2:** aligned length 24, odd length 19, and short length 3,
with both offload requests. Comparisons exercise actual attention and
loss backward, globally normalized loss, input/QKV/head gradients,
caller padding, and padding perturbations.
- Length 19 exercises both multi-chunk attention paths. Length 3 uses
the existing one-chunk non-offloading fallback, including when
offloading is requested.
- Changed-file pre-commit and strict FPDT F401 checks passed.
Environment: Python 3.11.10, Torch 2.11.0+cu128, CUDA 12.8, FA2 2.8.3,
pytest 8.3.5, NVIDIA driver 590.48.01.
New numerical coverage uses FP32 SDPA for Ulysses and FP16/SP2/zero
dropout for FPDT.
<details>
<summary>Test commands</summary>
```bash
# CPU environment without FA2
CUDA_VISIBLE_DEVICES='' DS_ACCELERATOR=cpu PYTHONPATH="$PWD" \
python -m pytest -q tests/unit/v1/sequence_parallelism/test_fpdt_inputs.py
# Full suite: four H200s
CUDA_VISIBLE_DEVICES=0,1,2,3 PYTHONPATH="$PWD" \
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \
python -m pytest --forked -q -rs \
tests/unit/v1/sequence_parallelism/test_fpdt_inputs.py \
tests/unit/v1/sequence_parallelism/test_ulysses.py \
tests/unit/ulysses_alst/test_ulysses_sp_hf.py --maxfail=1
```
</details>
The pre-existing `softmax().log()` instability is being addressed
separately in
[#9068](https://github.com/deepspeedai/DeepSpeed/pull/9068). This PR
documents zeroing losses at zero-weight positions before weighting,
avoiding `inf * 0 = NaN` for masked tokens.
## Duplicate Work
Addresses [#7384](https://github.com/deepspeedai/DeepSpeed/issues/7384)
and picks up the remaining work from
[#7868](https://github.com/deepspeedai/DeepSpeed/pull/7868).
[#8351](https://github.com/deepspeedai/DeepSpeed/pull/8351) proposed the
local divisibility assertion, this PR adds cross-rank validation and
FPDT padding. The sibling ALST data-loader adapter already rejects
non-divisible lengths and is unchanged. Our pre-submission search found
no other open implementation of this approach.
Builds on [@harshang03](https://github.com/harshang03)’s proposal and
[@PKUWZP](https://github.com/PKUWZP)’s review of
[#7868](https://github.com/deepspeedai/DeepSpeed/pull/7868).
AI assistance was used for implementation, testing, review, and
preparation of this PR.
---------
Signed-off-by: Jacob Schneider-Martin <jschneider@canva.com>