DeepSpeed
8702b42e - AutoEP: opt-in skip of the recomputed DeepEP combine (#8745)

Commit
3 days ago
AutoEP: opt-in skip of the recomputed DeepEP combine (#8745) ## Summary Add opt-in `expert_parallel.skip_recompute_combine` for AutoEP's DeepEP route. In supported Hugging Face per-decoder-layer **reentrant** checkpointing, the recomputed MoE output is used only by the residual add and is discarded by the checkpoint. The route records post-expert row weighting and combine backward without executing those forward operations; dispatch, expert compute and all required backward transfers still run. Default behavior is unchanged. This is the existing skip-combine PR, not a duplicate implementation. It remains distinct from the non-reentrant layout replay merged in deepspeedai/DeepSpeed#8731. The branch preserves that replay, the fused gate+up defaults from deepspeedai/DeepSpeed#8733, and the independent exact-size buffer checks from deepspeedai/DeepSpeed#8747. AI assistance was used for finalization, with author review before submission. ## Supported contract - `comm_backend="deepep"`, `autoep_size > 1`, BF16 dispatch; eager and fused row weighting, at their unchanged `score_apply` boundary. - Supported decoder presets: `deepseek_v2`, `deepseek_v3`, `minimax_m3`, `mixtral`, `qwen3_5_moe`, `qwen3_moe`, checked against Transformers 5.12.0 modeling code. - The recompute guard checks the attached decoder layer's current state and checkpoint configuration: `training=True`, `gradient_checkpointing=True` and the standard Torch checkpoint partial with `use_reentrant=True`. An eval-mode decoder does not run its own Hugging Face checkpoint, so an outer reentrant recompute is rejected rather than recording a NaN output. Ordinary eval forwards are unchanged. Transformers 5 defaults to non-reentrant, so explicitly use `gradient_checkpointing_kwargs={"use_reentrant": True}`. - Any non-null `moe_layer_pattern` is rejected during configuration validation when the skip is enabled, even with an explicit or automatically selected supported preset, a pattern matching the preset, or an empty string. Leave the pattern unset or disable the skip; custom-pattern support is unchanged with the option off. - Other checkpoint APIs, non-reentrant checkpointing with this flag and unsupported presets are outside the supported contract. Without recomputation, the ordinary path runs. - The skip requires that the MoE output has no value-dependent consumer in the recompute. The combined sentinel is NaN; an invalid consumer can produce non-finite values or gradients, but the sentinel is **not** a general runtime proof that arbitrary wrappers, hooks or patched decoder graphs are safe. ## Finalization Resolved the upstream conflicts without dropping replay or buffer behavior. Both normal and recorded combines now share their backward-context setup, including transferring the pending dispatch entry from the exchange to the consuming autograd node. Otherwise the recorded path could retain a stale entry and attempt to replay the previous step's routing in a later reentrant backward. The CPU reproducer fails before the context fix with `activation checkpointing recomputed a different routing`. The new regression performs three successive steps with changing routes and compares outputs, input/router/expert gradients and required transfers for both pre/post score application. The single CUDA weighting test now runs in a fresh `DistributedTest` worker, retaining its actual fused/recorded-backward comparison rather than allocating CUDA from an ordinary forked unittest. The existing real-DeepEP skip parity cases run two steps per model. The one-step test helper's defaults and original tolerances remain unchanged. ### Review follow-up Following the review, reject pattern overrides before model conversion. Applying an override preserves the supported preset name but can select a different decoder parent, so a preset-name whitelist cannot establish the residual-only output-consumer contract. The fix is a four-line configuration guard, not a decoder-graph validator or a runtime fallback. Six parametrized cases cover automatic and explicit preset selection, a custom parent, the preset's own pattern and an empty pattern. Each also checks that disabling the skip or leaving the pattern unset remains accepted. Both configuration references document the restriction. The kernels, recorded backward, supported execution paths and numerical thresholds are unchanged. ### Training-state guard follow-up The decoder must actually be checkpointing, not merely configured to checkpoint. Hugging Face enables the per-layer checkpoint only when `gradient_checkpointing` and `training` are both true. Without the training-state check, an eval decoder under an outer reentrant checkpoint can feed the recorded NaN into a value-dependent consumer such as `y * y`. The original guard was reproduced on CPU with finite forward output but non-finite input, router and expert gradients, for both pre and post score application. The guard now fails closed in that invalid recompute. A faithful decoder harness uses actual Torch nested reentrant checkpoints: training-mode output and every input/router/expert gradient match the unskipped CPU reference, eval-mode outer recomputes raise, and ordinary train/eval forwards retain the full route unchanged. The kernel, backward and numerical thresholds are unchanged. The latest master merge preserves the maintainers' existing merge commits and resolves only the import conflict by retaining `math`, `weakref` and `partial`. The lightweight `__new__` test layer explicitly initializes the optional overlap flag to its disabled default for eventual merge compatibility; the independent overlap feature from deepspeedai/DeepSpeed#8739 is not imported into this PR. ## Testing Latest training-state guard validation: **145 passed, 1 CUDA test skipped, 58 subtests passed**, on **1x Apple M4 Pro CPU**, macOS 26.7, Python 3.11.15, Torch **2.14.1**, pytest 9.1.1; CUDA and accelerator driver are not applicable. This covers the complete comm/replay suite, configuration tests and supported decoder attachment. The same selector previously reported **139 passed, 1 skipped, 52 subtests passed** for the earlier pattern-guard follow-up: ```bash DS_ACCELERATOR=cpu PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 PYTHONDONTWRITEBYTECODE=1 \ python -m pytest -q -p no:cacheprovider \ tests/unit/module_inject/test_auto_ep_comm.py \ tests/unit/v1/moe/test_autoep_unit.py::TestAutoEPConfig \ tests/unit/v1/moe/test_autoep_unit.py::TestModelDetectionAndReplacement::test_skip_recompute_combine_attaches_each_layer_to_its_decoder_layer ``` The skipped case requires CUDA and Triton for recorded fused weighting; its existing device evidence is listed below. The newly affected nested-checkpoint and ordinary-forward train/eval tests also pass under isolated forked workers (**2 passed, 0 skipped**): ```bash DS_ACCELERATOR=cpu PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 PYTHONDONTWRITEBYTECODE=1 \ python -m pytest -q -p no:cacheprovider -p pytest_forked --forked \ tests/unit/module_inject/test_auto_ep_comm.py::TestSkipRecomputeCombine::test_outer_recompute_respects_the_decoder_training_state \ tests/unit/module_inject/test_auto_ep_comm.py::TestSkipRecomputeCombine::test_nothing_is_skipped_without_a_recompute ``` The new regression produces exactly two missing-error failures with the original guard (eval mode, pre/post), while its two training controls pass. The earlier six pattern regressions passed under isolated forked workers (**6 passed, 0 skipped**) on the same CPU setup: ```bash DS_ACCELERATOR=cpu PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 PYTHONDONTWRITEBYTECODE=1 \ python -m pytest -q -p no:cacheprovider -p pytest_forked --forked \ tests/unit/v1/moe/test_autoep_unit.py::TestAutoEPConfig::test_skip_recompute_combine_rejects_pattern_overrides_even_with_supported_presets ``` Existing product validation is bound to the original PR commit `29303c378`: **275 passed, 1 skipped** on four H100s. All four eager/fused and uniform/skewed skip-parity cases ran, as did the recorded fused-weighting backward test. The skipped case is a MiniMax-M3 router-logit test requiring Transformers >=5.15.0; the validation environment has 5.12.0. This is reused evidence for the original implementation, not a claim that the entire suite was rerun after rebase. Pre-review focused CPU validation: **75 passed, 1 CUDA test skipped**, under both ordinary pytest and `pytest --forked`, on macOS with Torch 2.10.0 and Transformers 5.12.0. This covers the comm/replay suite, three skip config/conversion cases and the one-step helper defaults. Ordinary pytest also reports **52 subtests passed**. Earlier focused device validation, before the pattern and training-state guard follow-ups: **9 passed, 0 skipped** with `pytest --forked`, on **4x NVIDIA H100 80GB HBM3**, driver **580.159.04**, Torch **2.10.0.2+cu130 / CUDA 13.0**, Triton **3.6.0**, Transformers **5.12.0**, using the previously validated DeepEP runtime: - All four eager/fused and uniform/skewed skip-parity cases run two training steps on each model. They retain the original numerical bounds, check last-step output/input/router/parameter gradients and accumulated parameter deltas, and verify one forward combine per layer per step rather than two. - The four upstream non-reentrant/nested-reentrant replay cases still pass, confirming that the shared context preserves normal replay behavior. - The recorded fused-weighting backward comparison runs in a fresh worker. The original ordinary unittest reproduces `Cannot re-initialize CUDA in forked subprocess` on the same setup; the new worker-based case passes. GPU reproduction selector, from `tests/`: ```bash PYTHONPATH=.. DEEPSPEED_RUN_H100_TESTS=1 pytest --forked \ unit/module_inject/test_auto_ep_comm.py unit/v1/moe/test_autoep_deepep_parity.py \ -k 'skipping_the_recomputed_combine or recorded_fused_row_weighting or non_reentrant_checkpointing_replays' ``` Changed-files pre-commit passed for all 11 files in the final PR delta. Existing device evidence is reused for unchanged kernels/backward and supported training paths; GPU tests, full E2E and the existing performance matrix were not restarted for these guard follow-ups. The new state predicate is exercised through actual Torch checkpoint/autograd on CPU, not presented as a fresh H100 execution or a real Transformers-model run. The focused tests do not establish every ZeRO/offload/TP/PP combination, other hardware, arbitrary checkpoint composition or long-run convergence. The static CI selector includes the v1 AutoEP unit and parity files. Real DeepEP cases remain gated on the opt-in H100 test flag, four GPUs and a GIN-capable DeepEP build, so test collection alone does not prove they ran. The standalone recorded-weighting CUDA comparison is outside Modal's v1 test scope and was explicitly included in the focused hardware command above; its recorded path is also exercised by the v1 model-parity cases. Full GPU CI follows the repository's merge-queue workflow. ## Existing performance evidence Qwen3-30B-A3B, **8 H100s**, EP8, micro-batch **4**, gradient accumulation **16**, sequence **4096**, DeepEP **28 SMs**, reentrant activation checkpointing; **30 steps with 10 warmup**. The equivalent benchmark prototype records the same weighting/combine backward but returns uninitialized output where the product fills NaN. Both arms inherit concatenated gate+up and backward weight-gradient overlap; this is the **incremental skip effect**, not their combined benefit. One allocation, order normal / skip / skip / normal: | Recompute | Run medians | Mean | |---|---|---:| | Normal | 20,575.1 / 20,574.2 ms | 20,574.7 ms | | Combine + post-weighting forward skipped | 19,945.7 / 19,928.2 ms | **19,936.9 ms (-3.10%)** | Peak allocated memory stayed at **58.69 GB**; peak reserved memory was **0.25 GB lower**. Loss differences over these 30 steps were within same-arm repeat noise (maximum **9.4e-4**). The 8-GPU residual-block gate had numerical equality under `torch.equal` for output/input/router gradients and expert-gradient differences around **5e-5 relative L2**, within two normal recomputes' variation. `torch.equal` is not a claim about signed-zero bit patterns. The measurements are historical prototype evidence, **not** a retiming of the rebased product or a universal benefit across models/hardware. Skipping the forward combine removes that recompute transfer; it does not add overlap or remove the cached dispatch and weight-gradient communication required by backward. --------- Signed-off-by: yh0903 <helloyu0903@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: pengdurice <pengduhit@gmail.com>
Author
Parents
Loading