DeepSpeed
5dc44e84 - Overlap AutoEP's expert weight gradients with the DeepEP backward combine (#8739)

Commit
4 days ago
Overlap AutoEP's expert weight gradients with the DeepEP backward combine (#8739) The fused gate+up prerequisite, deepspeedai/DeepSpeed#8733, has merged. This branch is synchronized with current master and retains only the backward-overlap feature delta; no force-rewrite of the earlier submitted commits is required. In AutoEP's DeepEP backward, the experts compute their input gradient (dX) and their weight gradients (dW). DeepEP's combine then sends dX back to the token ranks. dW and the combine are independent, yet they run serially: a grouped GEMM's backward computes dX and dW together, and only then does the dispatch backward start the combine. In a kernel profile of Qwen3-30B-A3B on 8 H100s, DeepEP ran concurrently with no compute kernel at all. This adds an opt-in `expert_parallel.overlap_weight_grad`, which runs the combine and the dW GEMMs side by side. ### Change - `GroupedExperts.forward` takes an optional `ExpertWeightGradSlot`. - With a slot, `_DeferredWeightGradExperts` runs the same forward as the `torch._grouped_mm` path, for separate or fused gate+up and every registered activation. - Its backward returns dX at once and leaves the dW GEMMs in the slot. - The GEMMs are the products autograd forms for `torch._grouped_mm` (`_grouped_mm_mat1_backward`, `_grouped_mm_mat2_backward`). - The DeepEP dispatch takes the expert weights as extra inputs. Its backward: 1. launches the dX combine with `async_with_compute_stream=True`; 2. runs the slot's dW GEMMs on the compute stream; 3. waits on the combine's event; 4. returns dW through autograd, so ZeRO's gradient handling is unchanged. - SM carveout: - `torch._grouped_mm` launches persistent kernels sized to every SM, which leaves the combine nowhere to run. - The dW GEMMs are therefore launched with `comm_num_sm` SMs withheld through `torch._C._set_sm_carveout_experimental`. - The previous carveout is restored even if the GEMMs raise. - The pending combine's event is waited on in a `finally` block as well, so a dW error does not leave the queued communication unaccounted for. - Activation backward: - The activation registry gains optional standalone backward functions for the fused kernels. `swiglu` provides them, via new `swiglu_backward` and `swiglu_packed_backward`, which the existing autograd functions now also call. - Forms without them are recomputed and differentiated, so the deferred path always matches the forward that ran. - The default, off, keeps the existing path. - Master's per-layer checkpoint-dispatch replay and exact-size buffer-slice optimization are retained. Overlap slots are separate per invocation and do not replace replay handles. - The deferred custom backward is first-order only: `create_graph=True` raises rather than returning an incomplete derivative. - `overlap_weight_grad` is rejected, rather than ignored, unless all of these hold: - `comm_backend="deepep"` with `autoep_size > 1`; - `use_grouped_mm=true`, and the experts do not select the Triton grouped GEMM; - ZeRO stage 1 or 2, explicitly rejecting both stage 0 and stage 3; - PyTorch provides both the SM carveout getter and setter APIs. - Docs: `autoep.rst` and the config reference. ### Tests - CPU, `TestDeferredExpertWeightGrad`: - Output, input gradient and `w1`/`w2`/`w3` gradients match the plain path's values exactly in the tested CPU cases, for every activation, separate and fused gate+up, FP32 and BF16, with weight gradients immediate or deferred, including an empty expert. This does not assert universal bit-pattern equivalence. - Also: slot lifecycle errors, the `torch._grouped_mm` requirement, and config validation for every rejected setting. - CPU, `TestWeightGradOverlap`, the route against a DeepEP contract buffer: - Exact numerical agreement with the serial backward in the tested CPU cases, separate and fused gate+up, with no checkpointing, reentrant and non-reentrant checkpointing, now composed with upstream's replay coordinator. - The order is async combine, then dW with `comm_num_sm` withheld, then wait. - The carveout is restored, including on error. - A dW exception still waits for the async combine before propagating; it does not widen optimizer/backend support. - GPU, H100, 402 passed: - `test_overlapped_weight_gradients_match_the_serial_backward`, on 4 GPUs with DeepEP, uniform and skewed routing, separate and fused gate+up: input and every parameter gradient within 1e-2 relative L2, the shared parity checks, and no leaked carveout. - The CUDA bitwise check with the Triton kernels. - `swiglu_backward` and `swiglu_packed_backward` bitwise against autograd. - The rest of the DeepEP parity, AutoEP unit and activation suites. - Not part of this change: the 2 failures are `TestMiniMaxM3Block::test_replaced_block_matches_hf_block`. That preset needs Transformers >= 5.15.0, and the image has 5.12.0. - End to end, Qwen3-30B-A3B on 8 H100s: - Setup: expert parallel size 8, micro-batch 4, 16 accumulation steps, sequence length 4096, DeepEP, reentrant checkpointing, fused gate+up, 30 steps with 10 warmup. - Measured with an equivalent benchmark patch on our benchmark base: the same deferral, async combine and carveout. - Two pairs in one job: | Arm | Mean step | |---|---:| | Serial backward | 21,569.5 ms | | Overlap without the carveout | 21,418.5 ms (-0.70%) | | **Overlap with the carveout** | **21,097.3 ms (-2.19%)** | - A kernel profile showed DeepEP running alongside grouped GEMMs for 1,325 ms of the step, against none before. - Step-1 loss identical. Later losses agree within same-arm repeat noise. - Peak allocated memory unchanged; reserved memory about 0.7 GB higher. - A follow-up sweep in one job: withholding more SMs than DeepEP uses (24 with `comm_num_sm=12`) was 0.9% slower. That supports tying the carveout to `comm_num_sm`. - The overlap's gain depends on how much backward communication is left to hide. A second job separated the SM budget from the overlap: | Arm | Mean step | |---|---:| | Serial backward, `comm_num_sm=12` | 21,569.9 ms | | Serial backward, `comm_num_sm=20` | 20,942.6 ms | | **Overlap, `comm_num_sm=20`** | **20,696.9 ms (-1.17% vs serial at 20)** | | Overlap, `comm_num_sm=28` | 20,638.8 ms | On this single node, more DeepEP SMs speed up every DeepEP call regardless of this option. The overlap saves 472 ms at 12 SMs and 246 ms at 20. ### Review follow-up - The ZeRO-0 feedback is addressed by failing closed: `zero_stage not in (1, 2)` is rejected while the disabled default still permits the ordinary stage-0 path. - The engine now passes its actual ZeRO stage into that validator. Previously it omitted the keyword, so tightening the validator alone would incorrectly treat every optimizer as stage 0; both direct-config and engine-wiring cases cover stages 0/1/2/3. - Master synchronization preserves the already merged numerical-reference policy, packed-kernel input-device/stream guard and higher-order rejection. Extracted standalone backward helpers use the input device rather than whichever CUDA device happens to be current. - The real-DeepEP comparison now reads trainable accumulated gradients through `safe_get_full_grad` before `engine.step()`, including stage-2 cases where `param.grad` is released. It does not accept an empty expert-gradient dictionary as proof of parity. - Prior targeted H100 acceptance: **40 passed, 0 skipped**, on an 8x NVIDIA H100 80GB HBM3 host (distributed cases use four ranks), driver 580.159.04, torch 2.10.0.2/CUDA 13.0, Triton 3.6.0 and transformers 5.12.0. It covers both supported ZeRO stages, uniform/skewed routing, separate/fused gate+up, replay composition, standalone helper device selection and every trainable gradient before clipping. The 1e-2 relative-L2 criterion is unchanged. - On Apple M4 Pro CPU with torch 2.14.1, the comm suite passes **66/66 in forked mode**; targeted config/folding/activation checks pass. Removing only the three guards reproduces all **four** expected failures (ZeRO 0 accepted, missing event wait, and separate/fused incomplete higher-order derivatives). CPU mocks are not presented as physical DeepEP validation. - Existing 402-passing-case H100 and E2E results above are reused, not described as fresh broad-hardware or full-training validation. The new follow-up retains the old numerical thresholds and performs no throughput sweep. - The activation-weighting child, deepspeedai/DeepSpeed#8748, is reviewed after this parent and remains a separate opt-in feature; its shifted-rounding semantics are not part of this PR. AI assistance was used for review and test preparation. The human author reviewed the exact task-specific diff before signoff/publication. This updates an existing PR rather than opening a duplicate feature implementation. DCO, test collection and changed-files pre-commit passed on the published head. Full GPU CI and infrastructure smoke jobs were skipped; they are not described as a full GPU CI run. On-device evidence for this review follow-up is the targeted H100 execution above. ### Latest review follow-up Applied the requested minimal test-double repair: `record_validate` accepts and records `zero_stage`, and the existing sequence-parallel engine test asserts the actual stage is `0`. It no longer rejects the engine's keyword argument, and it would still catch missing stage forwarding instead of silently ignoring it. The original test reproduced `TypeError: ... unexpected keyword argument 'zero_stage'` before the repair. Synchronized with master to resolve the test-fixture import conflict. The worker-only fixture retains both this PR's standalone SwiGLU backward helpers and master's merged clamped activation helpers and tests. The only manual edits in this follow-up are in those two test files; no manual kernel, autograd, scheduling, numerical-threshold or default changes were made. The activation-weighting child remains separate. Validation on **1x Apple M4 Pro CPU**, macOS 26.7, Python 3.11.15, Torch **2.14.1**, pytest 9.1.1; CUDA and accelerator driver are not applicable: - Configuration, deferred expert gradients, ZeRO stage 0/1/2/3 engine wiring, and complete comm/replay coverage: **159 passed, 8 CUDA-only skipped, 49 subtests passed**. The repaired sequence-parallel test passes. - The repaired test also passes under `--forked`. Three selected SwiGLU cases reach the merged worker fixture and skip explicitly because CUDA/Triton is unavailable; this is import/worker coverage, not kernel execution. - All **155** SwiGLU cases collect, including both standalone backward and clamped forms. Changed-files pre-commit passes for all 14 files in the final PR delta. CPU reproduction command, from the repository root: ```bash DS_ACCELERATOR=cpu PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 PYTHONDONTWRITEBYTECODE=1 \ python -m pytest -q -p no:cacheprovider \ tests/unit/v1/moe/test_autoep_unit.py::TestAutoEPConfig \ tests/unit/v1/moe/test_autoep_expert_activation.py::TestDeferredExpertWeightGrad \ tests/unit/v1/moe/test_autoep_expert_activation.py::TestFusedGateUp::test_overlap_weight_grad_settings_are_validated \ tests/unit/v1/moe/test_autoep_expert_activation.py::TestFusedGateUp::test_overlap_weight_grad_rejects_the_triton_grouped_gemm \ tests/unit/v1/moe/test_autoep_expert_activation.py::TestFusedGateUp::test_engine_passes_the_actual_zero_stage_to_overlap_validation \ tests/unit/module_inject/test_auto_ep_comm.py ``` The prior H100 and E2E results above are reused evidence for the overlap implementation, not a new device run of this merged tree. GPU integration, throughput, memory and convergence experiments were not repeated for this test-only review repair and import-conflict resolution. No additional ZeRO/offload/TP/PP or hardware combinations are claimed. --------- Signed-off-by: yh0903 <helloyu0903@gmail.com> Signed-off-by: Masahiro Tanaka <tanaka.masahiro@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: pengdurice <pengduhit@gmail.com> Co-authored-by: Masahiro Tanaka <tanaka.masahiro@gmail.com>
Author
Parents
Loading