DeepSpeed
d70adc32 - [AutoEP] Stage expert weight gradients across microbatches (#9114)

Commit
1 day ago
[AutoEP] Stage expert weight gradients across microbatches (#9114) Implements the proposal in #9113: an opt-in AutoEP mode that stages the rows of the experts' weight-gradient GEMMs across microbatches and runs those GEMMs once per several backward passes. This PR was written with AI assistance (Claude Code). I have reviewed all of the code line by line and signed off each commit. ## What changes - `deepspeed/moe/ep_experts.py`: new `MoeDeferredDwStaging` (per-expert staging buffers and cursors; gradients are returned by one backward in every period and by the backward that closes an accumulation window; a full buffer runs the GEMMs early and holds the result until then). `_DeferredWeightGradExperts` and `GroupedExperts.forward` take an optional staging object. The dW products are factored into `_grouped_mlp_weight_grads`, shared by the immediate, slot and staged paths. - `deepspeed/module_inject/auto_ep_layer.py`, `auto_ep_config.py`, `auto_ep_presets/base.py`: `deferred_dw_flush_period` and `deferred_dw_rows_per_expert`, validation, and passing the staging object (and unpadded per-expert counts) at the three `self.experts(...)` call sites. - `deepspeed/runtime/engine.py` (core, wiring only): mark the backward that closes an accumulation window so it flushes; refuse an optimizer step with rows still staged; reject unmanaged accumulation, ZenFlow and DeepCompile. - `deepspeed/runtime/zero/stage_1_and_2.py` (core, 4 lines): the gradient hook fires even when the backward delivered no gradient for a parameter in the graph; `reduce_independent_p_g_buckets_and_remove_grads` now returns early in that case instead of dereferencing `None`. - Docs: `docs/_pages/config-json.md`, `docs/code-docs/source/autoep.rst`. ## Duplicate-work check Searched issues and PRs (open and closed) for "tgmm", "deferred dw", "MoeDeferredDwStaging", "weight gradient staging microbatch". The only related work is merged #8739 (`overlap_weight_grad`), which defers dW within a single backward to overlap it with the DeepEP combine; it does not accumulate rows across backward passes. This PR reuses its autograd function rather than adding a second one. Sibling paths checked: the same hook pattern exists in `zero/stage3.py`; it is not changed because this PR does not support ZeRO-3 (the stacked PR #9115 does it). The Triton grouped-GEMM path and the sequential expert loop are rejected by validation. ## Tests run Hardware: 3x NVIDIA H200 (143 GiB), driver 595.112.07, PyTorch 2.8.0+cu129, Python 3.12. `DS_SKIP_CUDA_CHECK=1` was set because the system CUDA toolkit (13.2) differs from PyTorch's. ``` cd tests pytest unit/v1/moe/test_autoep_expert_activation.py -k "TestDeferredDwStaging or TestPresetAndConfig or validated or rejected or rejects or test_slot or needs_the_grouped or create_graph or config_reaches" -> 49 passed, 2 skipped (1 GPU; run before the review fixes below) pytest unit/v1/moe/test_autoep_expert_activation.py -k TestDeferredDwStaging -> 36 passed (1 GPU, bf16; at 6ee0cbb9a) pytest unit/v1/moe/test_autoep_grad_parity.py -k TestAutoEPDeferredDwParity -> 6 passed (2 GPUs, comm backend, accumulation 4, flush period 3, two optimizer steps; at 6ee0cbb9a) expert parallel 2, ZeRO 0/1/2: staged vs. unstaged parity expert parallel 1 (both ranks replicate every expert, different data), ZeRO 1/2, 12 rows per expert: ranks flush early in different backwards and still reduce together pytest unit/v1/moe/test_autoep_expert_activation.py unit/module_inject/test_auto_ep_comm.py unit/v1/moe/test_autoep_unit.py -> 382 passed, 50 skipped (CPU, PyTorch 2.10.0+cpu, the version cpu-torch-latest installs; at 6ee0cbb9a) pytest unit/v1/moe/test_autoep_integration.py -> 7 passed, 3 skipped pytest unit/v1/zero/test_zero.py -k "TestZeroUnbalancedGradients or TestZeroFrozenWeights or TestZero3RepeatForwardLoop" -> 7 passed pytest unit/runtime/test_engine_attribute_delegation.py unit/compile/test_zero3_grad_dtype.py -> 22 passed pre-commit run --files <changed files> -> passed ``` The rows before `pre-commit` other than the first were run before the review fixes and not repeated; those fixes touch only `ep_experts.py`, docs and tests. Not validated: world size 4 with expert parallel 2 (the test machine has 3 GPUs; the 2-rank, expert-parallel-1 test has the same replicated-expert condition), DeepEP backend with staging, optimizer or parameter offload, AutoTP folding (rejected), ZeRO-3 (rejected here, #9115). A full GPU run of `test_autoep_expert_activation.py` hangs on this machine in `TestFusedGateUp::test_matches_separate_with_skewed_and_empty_experts`, which does not use staging, so only the selections above were run on GPU. Commit ids: the branch was rebased onto master `b9896d054` (a README-only change) after these runs. The results labelled 799bd4995 / 6ee0cbb9a were measured on the patch-identical commits 4954ced03 / ea50751bc before that rebase and were not repeated. ## Review fixes - 799bd4995: `test_auto_ep_comm.py` builds the layer without `__init__` (Mock stubs and `__new__`), so 14 tests lacked the new `dw_staging` / `_run_experts` and failed on cpu-torch-latest. Test-only. - 6ee0cbb9a (Codex P1, [thread](https://github.com/deepspeedai/DeepSpeed/pull/9114#discussion_r4621516055)): a full staging buffer made its rank return expert weight gradients early. Ranks holding replicas of the same experts fill their buffers in different backwards, and ZeRO-2 reduces in the backward that returns a gradient, so the replicas issued mismatched collectives. Reproduced as a hang with 2 ranks, expert parallel 1, ZeRO-2. An early flush now holds its result until the backward that closes the period or the window, so which backward returns gradients depends only on the count of backward passes. A period in which a rank's experts receive no rows now returns a zero gradient instead of none, for the same reason. ## Performance and memory Module-level only: grouped expert MLP forward+backward per microbatch on one H200, bf16, speedup vs. the same module without staging, flush period = accumulation window. | Shape (local experts, hidden, intermediate) | 8 rows/expert | 32 | 128 | 512 | | --- | --- | --- | --- | --- | | 128, 2048, 768, fused gate+up, window 8 | 1.69x | 1.62x | 1.34x | 0.90x | | 32, 7168, 2048, window 8 | 1.91x | 1.80x | 1.54x | 1.03x | | 512, 1024, 512, window 16 | 1.65x | 1.65x | 1.06x | 0.66x | Staging memory for the first shape at period 8: 0.20 / 0.78 / 3.13 / 12.5 GiB per layer per rank. It is a slowdown when experts receive many rows per backward. After a buffer overflows, the layer additionally holds one set of expert-weight-sized gradients until the next flush. These timings predate 6ee0cbb9a and were not repeated; without overflow the code path they measure is unchanged. Numerics: staged gradients match unstaged ones within bf16 GEMM rounding (max deviation below 1% of the largest gradient element in the engine runs). Against an fp64 reference the staged sum was closer (3.5e-3 vs 5.0e-3 to 6.1e-3 relative L2). ## Backward compatibility Off by default (`deferred_dw_flush_period: 1`). With it off, the only behavior change is the ZeRO-2 hook's early return for a `None` gradient, which previously raised `AttributeError`. --------- Signed-off-by: Zhipeng Wang <zhipeng.rainbowserie@gmail.com> Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Author
Parents
Loading