DeepSpeed
bfd29175 - AutoEP: reorder expert rows with gathers on the collective path (#8671)

Commit
48 days ago
AutoEP: reorder expert rows with gathers on the collective path (#8671) ## Summary AutoEP's native collective path (`comm_backend="comm"`, NCCL all-to-all) puts each rank's received rows into expert-major order before the grouped GEMM, and back afterwards, with `permute_by_local_expert` and `unpermute_by_local_expert`. The `autoep_size=1` path uses the same two functions. Both moved every routed row more often than needed: - **`permute_by_local_expert`:** - appended a zero row to the received rows with `torch.vstack`, a full copy, before indexing the padded expert-major buffer; - its backward was an accumulating `index_put`, i.e. a sort plus atomic adds. - **`unpermute_by_local_expert`:** zero-filled an `[N + 1, H]` buffer, then scattered into it with an elementwise `index_put`. On CUDA with Triton, both directions are now gathers: | Direction | Forward | Backward | |---|---|---| | permute | a Triton row gather that writes zero rows where `generate_permute_indices` marked padding (`-1`) | `index_select` through the inverse permutation | | unpermute | `index_select` through the inverse permutation | the same padded row gather | **Scope.** - This is a local, on-rank change to the expert reorder. - The all-to-alls send and receive the same rows in the same order, so communication volume is unchanged, and no communication/computation overlap is added. - `generate_permute_indices` is unchanged, and the DeepEP backend does not call these functions. - There is no switch: the gathers replace the previous code for 2-D CUDA tensors when Triton is available, because their results are equal to it (see Numerics). Other tensors keep the previous code. ## Permutation contract `generate_permute_indices` sends each received row to exactly one slot of the padded expert-major buffer and marks every other slot `-1`. The gathers rely on that one-to-one mapping: - **Permute forward:** writes each output row once: the row mapped to its slot, or zeros at a padding slot. - **Inverse permutation:** a scatter of slot positions into scratch of `N + N_padded` entries. Each padding slot goes to its own entry past `N`, so no entry is written twice, and every row's entry is written. - **Unpermute forward:** reads each row's slot through the inverse; padding rows are never read. - **Backward:** each direction's backward is the other gather. A row's gradient is that of the one slot it occupied, not a sum, and padding slots get zero gradient. Both backwards are autograd functions themselves, so double backward goes through the same pair. - **Top-k:** each received row is one (token, expert) assignment. Summing a token's top-k expert outputs, in the combine, and accumulating its top-k gradients, in the backward of the routed gather `x[token_indices // top_k]`, stay in the surrounding code, unchanged. These helpers are therefore not a general gather/scatter. `permute_rows`, `unpermute_rows` and `inverse_permutation` require the valid entries to be a complete one-to-one permutation of the rows, and do not check it; the docstrings say so. Both callers meet the requirement, because the counts sum to the number of rows: - with expert parallelism, each all-to-all receives exactly the row sums of the exchanged counts; - with `autoep_size=1`, the `T × top_k` routed rows are sorted by expert, and the counts are per expert. The previous code tolerated counts that sum to fewer rows (the rows left out came back as zeros); the gathers do not. The row gather reads its index by linear offset, so it now makes the index contiguous first, as it already did the source rows. ## Numerics The gathers only move data. Outputs, gradients and double-backward gradients equal the previous implementation's under `torch.equal`, which does not distinguish `+0.0` from `-0.0`. We also compared bit patterns on H100, in bf16, fp16 and fp32, with inputs and upstream gradients of which a quarter were `-0.0`: - Both outputs and the unpermute gradient are bitwise identical to the previous implementation's. - The permute gradient differs only where the upstream gradient is `-0.0`. The previous backward accumulated it into a zero-filled buffer, giving `+0.0`, whereas the gather passes `-0.0` through. ## Validation **Unit tests** (below): the earlier H100 validation passed these 27 tests and the existing `tests/unit/v1/moe/test_autoep_unit.py` (217 tests). The review follow-up additionally validates all 27 reorder cases with plain, forked and xdist-forked execution. **Local token movement**, one rank's work for one MoE layer, on one H100: - Shape: T tokens, top-8 of 128 experts, EP16 (8 local experts), hidden 2048. - The all-to-alls are replaced by the identity and the experts by a scalar multiply, so only the local work is timed. - Each iteration runs a no-grad forward, a grad forward and a backward, as under reentrant activation checkpointing. - The whole local chain is timed; only the reorder differs between the two columns. | T | Before | After | Saving × 48 layers | |---|---|---|---| | 8192 | 7.04 ms | 5.12 ms | −92 ms | | 16384 | 13.53 ms | 9.58 ms | −190 ms | At T=8192, per grad forward, the permute goes from 0.44 to 0.27 ms and the unpermute from 0.71 to 0.21 ms. **End to end**, Qwen3-30B-A3B training with AutoEP: - 16× H100, expert parallel 16, 48 layers, micro-batch 1, BF16, ZeRO stage 1, reentrant activation checkpointing, and a fixed routing, so both arms route identically. - `comm_backend="comm"` with `combine_impl="fused_weighted_sum"` and `async_split_plan=true`. - Both arms also carry other opt-in kernels under review: fused RMSNorm (#8645, #8664), fused RoPE (#8663), chunked loss (#8653) and BF16 gradient release (#8654). The arms differ only in this PR. - Both arms ran on the same 16 GPUs in mirrored order, two runs each. | Sequence | Before | After | Change | |---|---|---|---| | 8192 | 2649.4 ms | 2554.3 ms | −95.2 ms (−3.6%) | | 16384 | 5434.6 ms | 5244.5 ms | −190.2 ms (−3.5%) | - Each arm's two runs agree within 5 ms. - The end-to-end saving matches the local estimate scaled to 48 layers (−92 and −190 ms). - Peak memory and the step-0 loss are identical in both arms. These numbers come from one model, two sequence lengths, one GPU type and one parallel layout. The saving scales with the rows each rank reorders, so its share of the step depends on the configuration. Small shapes are not guaranteed to improve in every direction: a review microbenchmark on RTX PRO 4500 found slower forwards at 8,192 rows × hidden 1,025, while both backwards improved. This PR does not add a shape-dependent dispatch heuristic. The local and end-to-end numbers were measured with this PR's first commit. The later commits add `index.contiguous()`, a no-op for the contiguous indices both callers pass, and change docstrings and tests. The forked-runner review follow-up changes only test execution, not the runtime implementation. ## Tests `tests/unit/v1/moe/test_autoep_permute_rows.py`, 27 tests: 26 need CUDA and Triton, and 1 runs on the host. All compare against the previous zero-row-and-index implementation with `torch.equal`: - forward and backward of both reorders: - bf16 and fp32; - hidden 2048, 100 and 1; - source-major `[ep, E_local]` counts with and without empty experts, and aggregated `[E_local]` counts; - fp16 and bf16 rows read through a column-strided view, with column-major upstream gradients, in both directions, including the output strides; - double backward of both reorders; - a round trip with zero padding; - zero tokens; - the row gather with both padding encodings, through a contiguous and a strided index; - host tensors keeping the previous code. Results: - 1× NVIDIA H100 80GB HBM3, Python 3.12, torch 2.10 (CUDA 13.0), Triton 3.6.0: 27 passed, and 217 together with `tests/unit/v1/moe/test_autoep_unit.py`. - The same 27 tests against this PR's first commit, the code the timings used: 26 passed. The strided-index case failed, because that commit read the index as if contiguous. - CPU, torch 2.14: 1 passed, 26 skipped. - Forked-runner review regression on H100, torch 2.10 (CUDA 13.0), Triton 3.6.0: plain pytest, `pytest --forked` and `pytest -n 4 --forked` each pass all 27 cases, with no skips and no bad-fork errors. In the same environment, the previous tests fail under `--forked` (26 failed, 1 host case passed), reproducing the reported issue. - CPU review regression, torch 2.14: `--forked` and `-n 4 --forked` each give 1 passed, 26 skipped. - `pre-commit run --files` on the changed files: passed. ## CI - `cpu-torch-latest` runs the host test and skips the rest. - The test file is under `tests/unit/v1`, in the scope of both AWS and Modal GPU CI; Modal executes the selected GPU tests when the PR enters the merge queue. - `aws-torch-latest` uses `pytest -n 4 --forked`. The GPU cases now use `DistributedTest` with `world_size=1`, launching clean workers even when CUDA was touched while importing DeepSpeed during collection. These local operator tests do not need a process group (`init_distributed=False`). CUDA availability is checked by the runner during test execution, not by this file's collection-time skip condition. The host case remains an ordinary test, and all 27 cases and their numerical assertions are unchanged. ## Not covered - ROCm: PyTorch reports ROCm devices as `cuda`, so the gathers would be used there, as the existing Triton index generation is. Neither was tested on ROCm here. - Inputs that break the permutation contract. It is not checked; both callers meet it by construction. - `torch.compile` of the MoE layer itself. AutoEP's regional compilation keeps that layer eager, and the path already launched Triton kernels there. - Models, sequence lengths, hardware and parallel layouts other than the ones above. --------- Signed-off-by: yh0903 <helloyu0903@gmail.com> Co-authored-by: pengdurice <pengduhit@gmail.com>
Author
Parents
Loading