Overlap AutoEP's expert weight gradients with the DeepEP backward combine (#8739)
The fused gate+up prerequisite, deepspeedai/DeepSpeed#8733, has merged.
This branch is synchronized with current master and retains only the
backward-overlap feature delta; no force-rewrite of the earlier
submitted commits is required.
In AutoEP's DeepEP backward, the experts compute their input gradient
(dX) and their weight gradients (dW). DeepEP's combine then sends dX
back to the token ranks. dW and the combine are independent, yet they
run serially: a grouped GEMM's backward computes dX and dW together, and
only then does the dispatch backward start the combine. In a kernel
profile of Qwen3-30B-A3B on 8 H100s, DeepEP ran concurrently with no
compute kernel at all.
This adds an opt-in `expert_parallel.overlap_weight_grad`, which runs
the combine and the dW GEMMs side by side.
### Change
- `GroupedExperts.forward` takes an optional `ExpertWeightGradSlot`.
- With a slot, `_DeferredWeightGradExperts` runs the same forward as the
`torch._grouped_mm` path, for separate or fused gate+up and every
registered activation.
- Its backward returns dX at once and leaves the dW GEMMs in the slot.
- The GEMMs are the products autograd forms for `torch._grouped_mm`
(`_grouped_mm_mat1_backward`, `_grouped_mm_mat2_backward`).
- The DeepEP dispatch takes the expert weights as extra inputs. Its
backward:
1. launches the dX combine with `async_with_compute_stream=True`;
2. runs the slot's dW GEMMs on the compute stream;
3. waits on the combine's event;
4. returns dW through autograd, so ZeRO's gradient handling is
unchanged.
- SM carveout:
- `torch._grouped_mm` launches persistent kernels sized to every SM,
which leaves the combine nowhere to run.
- The dW GEMMs are therefore launched with `comm_num_sm` SMs withheld
through `torch._C._set_sm_carveout_experimental`.
- The previous carveout is restored even if the GEMMs raise.
- The pending combine's event is waited on in a `finally` block as well,
so a dW error does not leave the queued communication unaccounted for.
- Activation backward:
- The activation registry gains optional standalone backward functions
for the fused kernels. `swiglu` provides them, via new `swiglu_backward`
and `swiglu_packed_backward`, which the existing autograd functions now
also call.
- Forms without them are recomputed and differentiated, so the deferred
path always matches the forward that ran.
- The default, off, keeps the existing path.
- Master's per-layer checkpoint-dispatch replay and exact-size
buffer-slice optimization are retained. Overlap slots are separate per
invocation and do not replace replay handles.
- The deferred custom backward is first-order only: `create_graph=True`
raises rather than returning an incomplete derivative.
- `overlap_weight_grad` is rejected, rather than ignored, unless all of
these hold:
- `comm_backend="deepep"` with `autoep_size > 1`;
- `use_grouped_mm=true`, and the experts do not select the Triton
grouped GEMM;
- ZeRO stage 1 or 2, explicitly rejecting both stage 0 and stage 3;
- PyTorch provides both the SM carveout getter and setter APIs.
- Docs: `autoep.rst` and the config reference.
### Tests
- CPU, `TestDeferredExpertWeightGrad`:
- Output, input gradient and `w1`/`w2`/`w3` gradients match the plain
path's values exactly in the tested CPU cases, for every activation,
separate and fused gate+up, FP32 and BF16, with weight gradients
immediate or deferred, including an empty expert. This does not assert
universal bit-pattern equivalence.
- Also: slot lifecycle errors, the `torch._grouped_mm` requirement, and
config validation for every rejected setting.
- CPU, `TestWeightGradOverlap`, the route against a DeepEP contract
buffer:
- Exact numerical agreement with the serial backward in the tested CPU
cases, separate and fused gate+up, with no checkpointing, reentrant and
non-reentrant checkpointing, now composed with upstream's replay
coordinator.
- The order is async combine, then dW with `comm_num_sm` withheld, then
wait.
- The carveout is restored, including on error.
- A dW exception still waits for the async combine before propagating;
it does not widen optimizer/backend support.
- GPU, H100, 402 passed:
- `test_overlapped_weight_gradients_match_the_serial_backward`, on 4
GPUs with DeepEP, uniform and skewed routing, separate and fused
gate+up: input and every parameter gradient within 1e-2 relative L2, the
shared parity checks, and no leaked carveout.
- The CUDA bitwise check with the Triton kernels.
- `swiglu_backward` and `swiglu_packed_backward` bitwise against
autograd.
- The rest of the DeepEP parity, AutoEP unit and activation suites.
- Not part of this change: the 2 failures are
`TestMiniMaxM3Block::test_replaced_block_matches_hf_block`. That preset
needs Transformers >= 5.15.0, and the image has 5.12.0.
- End to end, Qwen3-30B-A3B on 8 H100s:
- Setup: expert parallel size 8, micro-batch 4, 16 accumulation steps,
sequence length 4096, DeepEP, reentrant checkpointing, fused gate+up, 30
steps with 10 warmup.
- Measured with an equivalent benchmark patch on our benchmark base: the
same deferral, async combine and carveout.
- Two pairs in one job:
| Arm | Mean step |
|---|---:|
| Serial backward | 21,569.5 ms |
| Overlap without the carveout | 21,418.5 ms (-0.70%) |
| **Overlap with the carveout** | **21,097.3 ms (-2.19%)** |
- A kernel profile showed DeepEP running alongside grouped GEMMs for
1,325 ms of the step, against none before.
- Step-1 loss identical. Later losses agree within same-arm repeat
noise.
- Peak allocated memory unchanged; reserved memory about 0.7 GB higher.
- A follow-up sweep in one job: withholding more SMs than DeepEP uses
(24 with `comm_num_sm=12`) was 0.9% slower. That supports tying the
carveout to `comm_num_sm`.
- The overlap's gain depends on how much backward communication is left
to hide. A second job separated the SM budget from the overlap:
| Arm | Mean step |
|---|---:|
| Serial backward, `comm_num_sm=12` | 21,569.9 ms |
| Serial backward, `comm_num_sm=20` | 20,942.6 ms |
| **Overlap, `comm_num_sm=20`** | **20,696.9 ms (-1.17% vs serial at
20)** |
| Overlap, `comm_num_sm=28` | 20,638.8 ms |
On this single node, more DeepEP SMs speed up every DeepEP call
regardless of this option. The overlap saves 472 ms at 12 SMs and 246 ms
at 20.
### Review follow-up
- The ZeRO-0 feedback is addressed by failing closed: `zero_stage not in
(1, 2)` is rejected while the disabled default still permits the
ordinary stage-0 path.
- The engine now passes its actual ZeRO stage into that validator.
Previously it omitted the keyword, so tightening the validator alone
would incorrectly treat every optimizer as stage 0; both direct-config
and engine-wiring cases cover stages 0/1/2/3.
- Master synchronization preserves the already merged
numerical-reference policy, packed-kernel input-device/stream guard and
higher-order rejection. Extracted standalone backward helpers use the
input device rather than whichever CUDA device happens to be current.
- The real-DeepEP comparison now reads trainable accumulated gradients
through `safe_get_full_grad` before `engine.step()`, including stage-2
cases where `param.grad` is released. It does not accept an empty
expert-gradient dictionary as proof of parity.
- Prior targeted H100 acceptance: **40 passed, 0 skipped**, on an 8x
NVIDIA H100 80GB HBM3 host (distributed cases use four ranks), driver
580.159.04, torch 2.10.0.2/CUDA 13.0, Triton 3.6.0 and transformers
5.12.0. It covers both supported ZeRO stages, uniform/skewed routing,
separate/fused gate+up, replay composition, standalone helper device
selection and every trainable gradient before clipping. The 1e-2
relative-L2 criterion is unchanged.
- On Apple M4 Pro CPU with torch 2.14.1, the comm suite passes **66/66
in forked mode**; targeted config/folding/activation checks pass.
Removing only the three guards reproduces all **four** expected failures
(ZeRO 0 accepted, missing event wait, and separate/fused incomplete
higher-order derivatives). CPU mocks are not presented as physical
DeepEP validation.
- Existing 402-passing-case H100 and E2E results above are reused, not
described as fresh broad-hardware or full-training validation. The new
follow-up retains the old numerical thresholds and performs no
throughput sweep.
- The activation-weighting child, deepspeedai/DeepSpeed#8748, is
reviewed after this parent and remains a separate opt-in feature; its
shifted-rounding semantics are not part of this PR.
AI assistance was used for review and test preparation. The human author
reviewed the exact task-specific diff before signoff/publication. This
updates an existing PR rather than opening a duplicate feature
implementation.
DCO, test collection and changed-files pre-commit passed on the
published head. Full GPU CI and infrastructure smoke jobs were skipped;
they are not described as a full GPU CI run. On-device evidence for this
review follow-up is the targeted H100 execution above.
### Latest review follow-up
Applied the requested minimal test-double repair: `record_validate`
accepts and records `zero_stage`, and the existing sequence-parallel
engine test asserts the actual stage is `0`. It no longer rejects the
engine's keyword argument, and it would still catch missing stage
forwarding instead of silently ignoring it. The original test reproduced
`TypeError: ... unexpected keyword argument 'zero_stage'` before the
repair.
Synchronized with master to resolve the test-fixture import conflict.
The worker-only fixture retains both this PR's standalone SwiGLU
backward helpers and master's merged clamped activation helpers and
tests. The only manual edits in this follow-up are in those two test
files; no manual kernel, autograd, scheduling, numerical-threshold or
default changes were made. The activation-weighting child remains
separate.
Validation on **1x Apple M4 Pro CPU**, macOS 26.7, Python 3.11.15, Torch
**2.14.1**, pytest 9.1.1; CUDA and accelerator driver are not
applicable:
- Configuration, deferred expert gradients, ZeRO stage 0/1/2/3 engine
wiring, and complete comm/replay coverage: **159 passed, 8 CUDA-only
skipped, 49 subtests passed**. The repaired sequence-parallel test
passes.
- The repaired test also passes under `--forked`. Three selected SwiGLU
cases reach the merged worker fixture and skip explicitly because
CUDA/Triton is unavailable; this is import/worker coverage, not kernel
execution.
- All **155** SwiGLU cases collect, including both standalone backward
and clamped forms. Changed-files pre-commit passes for all 14 files in
the final PR delta.
CPU reproduction command, from the repository root:
```bash
DS_ACCELERATOR=cpu PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 PYTHONDONTWRITEBYTECODE=1 \
python -m pytest -q -p no:cacheprovider \
tests/unit/v1/moe/test_autoep_unit.py::TestAutoEPConfig \
tests/unit/v1/moe/test_autoep_expert_activation.py::TestDeferredExpertWeightGrad \
tests/unit/v1/moe/test_autoep_expert_activation.py::TestFusedGateUp::test_overlap_weight_grad_settings_are_validated \
tests/unit/v1/moe/test_autoep_expert_activation.py::TestFusedGateUp::test_overlap_weight_grad_rejects_the_triton_grouped_gemm \
tests/unit/v1/moe/test_autoep_expert_activation.py::TestFusedGateUp::test_engine_passes_the_actual_zero_stage_to_overlap_validation \
tests/unit/module_inject/test_auto_ep_comm.py
```
The prior H100 and E2E results above are reused evidence for the overlap
implementation, not a new device run of this merged tree. GPU
integration, throughput, memory and convergence experiments were not
repeated for this test-only review repair and import-conflict
resolution. No additional ZeRO/offload/TP/PP or hardware combinations
are claimed.
---------
Signed-off-by: yh0903 <helloyu0903@gmail.com>
Signed-off-by: Masahiro Tanaka <tanaka.masahiro@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: pengdurice <pengduhit@gmail.com>
Co-authored-by: Masahiro Tanaka <tanaka.masahiro@gmail.com>