[AutoEP] Stage expert weight gradients across microbatches (#9114)
Implements the proposal in #9113: an opt-in AutoEP mode that stages the
rows of the experts' weight-gradient GEMMs across microbatches and runs
those GEMMs once per several backward passes.
This PR was written with AI assistance (Claude Code). I have reviewed
all of the code line by line and signed off each commit.
## What changes
- `deepspeed/moe/ep_experts.py`: new `MoeDeferredDwStaging` (per-expert
staging buffers and cursors; gradients are returned by one backward in
every period and by the backward that closes an accumulation window; a
full buffer runs the GEMMs early and holds the result until then).
`_DeferredWeightGradExperts` and `GroupedExperts.forward` take an
optional staging object. The dW products are factored into
`_grouped_mlp_weight_grads`, shared by the immediate, slot and staged
paths.
- `deepspeed/module_inject/auto_ep_layer.py`, `auto_ep_config.py`,
`auto_ep_presets/base.py`: `deferred_dw_flush_period` and
`deferred_dw_rows_per_expert`, validation, and passing the staging
object (and unpadded per-expert counts) at the three `self.experts(...)`
call sites.
- `deepspeed/runtime/engine.py` (core, wiring only): mark the backward
that closes an accumulation window so it flushes; refuse an optimizer
step with rows still staged; reject unmanaged accumulation, ZenFlow and
DeepCompile.
- `deepspeed/runtime/zero/stage_1_and_2.py` (core, 4 lines): the
gradient hook fires even when the backward delivered no gradient for a
parameter in the graph;
`reduce_independent_p_g_buckets_and_remove_grads` now returns early in
that case instead of dereferencing `None`.
- Docs: `docs/_pages/config-json.md`,
`docs/code-docs/source/autoep.rst`.
## Duplicate-work check
Searched issues and PRs (open and closed) for "tgmm", "deferred dw",
"MoeDeferredDwStaging", "weight gradient staging microbatch". The only
related work is merged #8739 (`overlap_weight_grad`), which defers dW
within a single backward to overlap it with the DeepEP combine; it does
not accumulate rows across backward passes. This PR reuses its autograd
function rather than adding a second one.
Sibling paths checked: the same hook pattern exists in `zero/stage3.py`;
it is not changed because this PR does not support ZeRO-3 (the stacked
PR #9115 does it). The Triton grouped-GEMM path and the sequential
expert loop are rejected by validation.
## Tests run
Hardware: 3x NVIDIA H200 (143 GiB), driver 595.112.07, PyTorch
2.8.0+cu129, Python 3.12. `DS_SKIP_CUDA_CHECK=1` was set because the
system CUDA toolkit (13.2) differs from PyTorch's.
```
cd tests
pytest unit/v1/moe/test_autoep_expert_activation.py -k "TestDeferredDwStaging or TestPresetAndConfig or validated or rejected or rejects or test_slot or needs_the_grouped or create_graph or config_reaches"
-> 49 passed, 2 skipped (1 GPU; run before the review fixes below)
pytest unit/v1/moe/test_autoep_expert_activation.py -k TestDeferredDwStaging
-> 36 passed (1 GPU, bf16; at 6ee0cbb9a)
pytest unit/v1/moe/test_autoep_grad_parity.py -k TestAutoEPDeferredDwParity
-> 6 passed (2 GPUs, comm backend, accumulation 4, flush period 3, two optimizer steps; at 6ee0cbb9a)
expert parallel 2, ZeRO 0/1/2: staged vs. unstaged parity
expert parallel 1 (both ranks replicate every expert, different data), ZeRO 1/2, 12 rows per expert:
ranks flush early in different backwards and still reduce together
pytest unit/v1/moe/test_autoep_expert_activation.py unit/module_inject/test_auto_ep_comm.py unit/v1/moe/test_autoep_unit.py
-> 382 passed, 50 skipped (CPU, PyTorch 2.10.0+cpu, the version cpu-torch-latest installs; at 6ee0cbb9a)
pytest unit/v1/moe/test_autoep_integration.py
-> 7 passed, 3 skipped
pytest unit/v1/zero/test_zero.py -k "TestZeroUnbalancedGradients or TestZeroFrozenWeights or TestZero3RepeatForwardLoop"
-> 7 passed
pytest unit/runtime/test_engine_attribute_delegation.py unit/compile/test_zero3_grad_dtype.py
-> 22 passed
pre-commit run --files <changed files>
-> passed
```
The rows before `pre-commit` other than the first were run before the
review fixes and not repeated; those fixes touch only `ep_experts.py`,
docs and tests.
Not validated: world size 4 with expert parallel 2 (the test machine has
3 GPUs; the 2-rank, expert-parallel-1 test has the same
replicated-expert condition), DeepEP backend with staging, optimizer or
parameter offload, AutoTP folding (rejected), ZeRO-3 (rejected here,
#9115). A full GPU run of `test_autoep_expert_activation.py` hangs on
this machine in
`TestFusedGateUp::test_matches_separate_with_skewed_and_empty_experts`,
which does not use staging, so only the selections above were run on
GPU.
Commit ids: the branch was rebased onto master `b9896d054` (a
README-only change) after these runs. The results labelled 799bd4995 /
6ee0cbb9a were measured on the patch-identical commits 4954ced03 /
ea50751bc before that rebase and were not repeated.
## Review fixes
- 799bd4995: `test_auto_ep_comm.py` builds the layer without `__init__`
(Mock stubs and `__new__`), so 14 tests lacked the new `dw_staging` /
`_run_experts` and failed on cpu-torch-latest. Test-only.
- 6ee0cbb9a (Codex P1,
[thread](https://github.com/deepspeedai/DeepSpeed/pull/9114#discussion_r4621516055)):
a full staging buffer made its rank return expert weight gradients
early. Ranks holding replicas of the same experts fill their buffers in
different backwards, and ZeRO-2 reduces in the backward that returns a
gradient, so the replicas issued mismatched collectives. Reproduced as a
hang with 2 ranks, expert parallel 1, ZeRO-2. An early flush now holds
its result until the backward that closes the period or the window, so
which backward returns gradients depends only on the count of backward
passes. A period in which a rank's experts receive no rows now returns a
zero gradient instead of none, for the same reason.
## Performance and memory
Module-level only: grouped expert MLP forward+backward per microbatch on
one H200, bf16, speedup vs. the same module without staging, flush
period = accumulation window.
| Shape (local experts, hidden, intermediate) | 8 rows/expert | 32 | 128
| 512 |
| --- | --- | --- | --- | --- |
| 128, 2048, 768, fused gate+up, window 8 | 1.69x | 1.62x | 1.34x |
0.90x |
| 32, 7168, 2048, window 8 | 1.91x | 1.80x | 1.54x | 1.03x |
| 512, 1024, 512, window 16 | 1.65x | 1.65x | 1.06x | 0.66x |
Staging memory for the first shape at period 8: 0.20 / 0.78 / 3.13 /
12.5 GiB per layer per rank. It is a slowdown when experts receive many
rows per backward. After a buffer overflows, the layer additionally
holds one set of expert-weight-sized gradients until the next flush.
These timings predate 6ee0cbb9a and were not repeated; without overflow
the code path they measure is unchanged.
Numerics: staged gradients match unstaged ones within bf16 GEMM rounding
(max deviation below 1% of the largest gradient element in the engine
runs). Against an fp64 reference the staged sum was closer (3.5e-3 vs
5.0e-3 to 6.1e-3 relative L2).
## Backward compatibility
Off by default (`deferred_dw_flush_period: 1`). With it off, the only
behavior change is the ZeRO-2 hook's early return for a `None` gradient,
which previously raised `AttributeError`.
---------
Signed-off-by: Zhipeng Wang <zhipeng.rainbowserie@gmail.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>