AutoEP: reorder expert rows with gathers on the collective path (#8671)
## Summary
AutoEP's native collective path (`comm_backend="comm"`, NCCL all-to-all)
puts each rank's received rows into expert-major order before the
grouped GEMM, and back afterwards, with `permute_by_local_expert` and
`unpermute_by_local_expert`. The `autoep_size=1` path uses the same two
functions. Both moved every routed row more often than needed:
- **`permute_by_local_expert`:**
- appended a zero row to the received rows with `torch.vstack`, a full
copy, before indexing the padded expert-major buffer;
- its backward was an accumulating `index_put`, i.e. a sort plus atomic
adds.
- **`unpermute_by_local_expert`:** zero-filled an `[N + 1, H]` buffer,
then scattered into it with an elementwise `index_put`.
On CUDA with Triton, both directions are now gathers:
| Direction | Forward | Backward |
|---|---|---|
| permute | a Triton row gather that writes zero rows where
`generate_permute_indices` marked padding (`-1`) | `index_select`
through the inverse permutation |
| unpermute | `index_select` through the inverse permutation | the same
padded row gather |
**Scope.**
- This is a local, on-rank change to the expert reorder.
- The all-to-alls send and receive the same rows in the same order, so
communication volume is unchanged, and no communication/computation
overlap is added.
- `generate_permute_indices` is unchanged, and the DeepEP backend does
not call these functions.
- There is no switch: the gathers replace the previous code for 2-D CUDA
tensors when Triton is available, because their results are equal to it
(see Numerics). Other tensors keep the previous code.
## Permutation contract
`generate_permute_indices` sends each received row to exactly one slot
of the padded expert-major buffer and marks every other slot `-1`. The
gathers rely on that one-to-one mapping:
- **Permute forward:** writes each output row once: the row mapped to
its slot, or zeros at a padding slot.
- **Inverse permutation:** a scatter of slot positions into scratch of
`N + N_padded` entries. Each padding slot goes to its own entry past
`N`, so no entry is written twice, and every row's entry is written.
- **Unpermute forward:** reads each row's slot through the inverse;
padding rows are never read.
- **Backward:** each direction's backward is the other gather. A row's
gradient is that of the one slot it occupied, not a sum, and padding
slots get zero gradient. Both backwards are autograd functions
themselves, so double backward goes through the same pair.
- **Top-k:** each received row is one (token, expert) assignment.
Summing a token's top-k expert outputs, in the combine, and accumulating
its top-k gradients, in the backward of the routed gather
`x[token_indices // top_k]`, stay in the surrounding code, unchanged.
These helpers are therefore not a general gather/scatter.
`permute_rows`, `unpermute_rows` and `inverse_permutation` require the
valid entries to be a complete one-to-one permutation of the rows, and
do not check it; the docstrings say so.
Both callers meet the requirement, because the counts sum to the number
of rows:
- with expert parallelism, each all-to-all receives exactly the row sums
of the exchanged counts;
- with `autoep_size=1`, the `T × top_k` routed rows are sorted by
expert, and the counts are per expert.
The previous code tolerated counts that sum to fewer rows (the rows left
out came back as zeros); the gathers do not. The row gather reads its
index by linear offset, so it now makes the index contiguous first, as
it already did the source rows.
## Numerics
The gathers only move data. Outputs, gradients and double-backward
gradients equal the previous implementation's under `torch.equal`, which
does not distinguish `+0.0` from `-0.0`.
We also compared bit patterns on H100, in bf16, fp16 and fp32, with
inputs and upstream gradients of which a quarter were `-0.0`:
- Both outputs and the unpermute gradient are bitwise identical to the
previous implementation's.
- The permute gradient differs only where the upstream gradient is
`-0.0`. The previous backward accumulated it into a zero-filled buffer,
giving `+0.0`, whereas the gather passes `-0.0` through.
## Validation
**Unit tests** (below): the earlier H100 validation passed these 27
tests and the existing `tests/unit/v1/moe/test_autoep_unit.py` (217
tests). The review follow-up additionally validates all 27 reorder cases
with plain, forked and xdist-forked execution.
**Local token movement**, one rank's work for one MoE layer, on one
H100:
- Shape: T tokens, top-8 of 128 experts, EP16 (8 local experts), hidden
2048.
- The all-to-alls are replaced by the identity and the experts by a
scalar multiply, so only the local work is timed.
- Each iteration runs a no-grad forward, a grad forward and a backward,
as under reentrant activation checkpointing.
- The whole local chain is timed; only the reorder differs between the
two columns.
| T | Before | After | Saving × 48 layers |
|---|---|---|---|
| 8192 | 7.04 ms | 5.12 ms | −92 ms |
| 16384 | 13.53 ms | 9.58 ms | −190 ms |
At T=8192, per grad forward, the permute goes from 0.44 to 0.27 ms and
the unpermute from 0.71 to 0.21 ms.
**End to end**, Qwen3-30B-A3B training with AutoEP:
- 16× H100, expert parallel 16, 48 layers, micro-batch 1, BF16, ZeRO
stage 1, reentrant activation checkpointing, and a fixed routing, so
both arms route identically.
- `comm_backend="comm"` with `combine_impl="fused_weighted_sum"` and
`async_split_plan=true`.
- Both arms also carry other opt-in kernels under review: fused RMSNorm
(#8645, #8664), fused RoPE (#8663), chunked loss (#8653) and BF16
gradient release (#8654). The arms differ only in this PR.
- Both arms ran on the same 16 GPUs in mirrored order, two runs each.
| Sequence | Before | After | Change |
|---|---|---|---|
| 8192 | 2649.4 ms | 2554.3 ms | −95.2 ms (−3.6%) |
| 16384 | 5434.6 ms | 5244.5 ms | −190.2 ms (−3.5%) |
- Each arm's two runs agree within 5 ms.
- The end-to-end saving matches the local estimate scaled to 48 layers
(−92 and −190 ms).
- Peak memory and the step-0 loss are identical in both arms.
These numbers come from one model, two sequence lengths, one GPU type
and one parallel layout. The saving scales with the rows each rank
reorders, so its share of the step depends on the configuration.
Small shapes are not guaranteed to improve in every direction: a review
microbenchmark on RTX PRO 4500 found slower forwards at 8,192 rows ×
hidden 1,025, while both backwards improved. This PR does not add a
shape-dependent dispatch heuristic.
The local and end-to-end numbers were measured with this PR's first
commit. The later commits add `index.contiguous()`, a no-op for the
contiguous indices both callers pass, and change docstrings and tests.
The forked-runner review follow-up changes only test execution, not the
runtime implementation.
## Tests
`tests/unit/v1/moe/test_autoep_permute_rows.py`, 27 tests: 26 need CUDA
and Triton, and 1 runs on the host. All compare against the previous
zero-row-and-index implementation with `torch.equal`:
- forward and backward of both reorders:
- bf16 and fp32;
- hidden 2048, 100 and 1;
- source-major `[ep, E_local]` counts with and without empty experts,
and aggregated `[E_local]` counts;
- fp16 and bf16 rows read through a column-strided view, with
column-major upstream gradients, in both directions, including the
output strides;
- double backward of both reorders;
- a round trip with zero padding;
- zero tokens;
- the row gather with both padding encodings, through a contiguous and a
strided index;
- host tensors keeping the previous code.
Results:
- 1× NVIDIA H100 80GB HBM3, Python 3.12, torch 2.10 (CUDA 13.0), Triton
3.6.0: 27 passed, and 217 together with
`tests/unit/v1/moe/test_autoep_unit.py`.
- The same 27 tests against this PR's first commit, the code the timings
used: 26 passed. The strided-index case failed, because that commit read
the index as if contiguous.
- CPU, torch 2.14: 1 passed, 26 skipped.
- Forked-runner review regression on H100, torch 2.10 (CUDA 13.0),
Triton 3.6.0: plain pytest, `pytest --forked` and `pytest -n 4 --forked`
each pass all 27 cases, with no skips and no bad-fork errors. In the
same environment, the previous tests fail under `--forked` (26 failed, 1
host case passed), reproducing the reported issue.
- CPU review regression, torch 2.14: `--forked` and `-n 4 --forked` each
give 1 passed, 26 skipped.
- `pre-commit run --files` on the changed files: passed.
## CI
- `cpu-torch-latest` runs the host test and skips the rest.
- The test file is under `tests/unit/v1`, in the scope of both AWS and
Modal GPU CI; Modal executes the selected GPU tests when the PR enters
the merge queue.
- `aws-torch-latest` uses `pytest -n 4 --forked`. The GPU cases now use
`DistributedTest` with `world_size=1`, launching clean workers even when
CUDA was touched while importing DeepSpeed during collection. These
local operator tests do not need a process group
(`init_distributed=False`). CUDA availability is checked by the runner
during test execution, not by this file's collection-time skip
condition. The host case remains an ordinary test, and all 27 cases and
their numerical assertions are unchanged.
## Not covered
- ROCm: PyTorch reports ROCm devices as `cuda`, so the gathers would be
used there, as the existing Triton index generation is. Neither was
tested on ROCm here.
- Inputs that break the permutation contract. It is not checked; both
callers meet it by construction.
- `torch.compile` of the MoE layer itself. AutoEP's regional compilation
keeps that layer eager, and the path already launched Triton kernels
there.
- Models, sequence lengths, hardware and parallel layouts other than the
ones above.
---------
Signed-off-by: yh0903 <helloyu0903@gmail.com>
Co-authored-by: pengdurice <pengduhit@gmail.com>