DeepSpeed
17828f6e - Universal Checkpointing: read AutoEP ZeRO-3 optimizer files one at a time (#8844)

Commit
1 day ago
Universal Checkpointing: read AutoEP ZeRO-3 optimizer files one at a time (#8844) # Universal Checkpointing: read AutoEP ZeRO-3 optimizer files one at a time ## Problem `ds_to_universal` converts a ZeRO-3 AutoEP checkpoint in two parts. The dense parameters go through the usual extract-and-merge path, which reads one optimizer file per worker and writes fragments to `tmp/`. The routed experts go through `_consolidate_zero3_autoep_expert_states`, which first loads **every** rank's optimizer file into one dict (`optim_states_by_rank`) and then keeps a copy of every expert fragment in memory until all of them are merged. Together those files hold every optimizer state of the model: fp32 (or bf16) master weights plus both Adam moments. For a large MoE model that is more than one host has: | Model | Parameters | Optimizer state, fp32 master + fp32 Adam (12 B/param) | bf16 master + bf16 Adam (6 B/param) | |---|---|---|---| | MiniMax-M3 | 426.2B | 4.65 TiB | 2.33 TiB | | GLM-5.2 | 743.4B | 8.11 TiB | 4.06 TiB | The H200 nodes this was found on have 2 TiB of host memory, so the conversion cannot finish for either model. ## Change `_consolidate_zero3_autoep_expert_states` reads one rank's optimizer file at a time: - Each expert fragment it finds is written to `<output>/tmp/autoep_expert_fragments/` (one file per parameter, optimizer state, EP rank and partition rank). - Every reference into the rank's optimizer state, including the loop variables, is dropped before the next file is read. - The merge then loads, for one parameter and one state at a time, only that parameter's fragments, and deletes each fragment file once it is read, so the disk does not hold every fragment and every universal file at the same time. Peak host memory becomes one optimizer file plus one full expert tensor, instead of all optimizer files. The checks are unchanged: missing ranks, duplicate or missing expert-data-parallel partitions, shape mismatches and placement disagreements raise the same errors. The temporary fragments live under `<output>/tmp`, which the converter already uses for the dense fragments and removes at the end. ## Peak host memory Measured with a probe on one checkpoint: a 4-layer mock MoE model (64 experts, hidden 1024, expert intermediate 4096: 3.22B expert parameters) trained one step and saved with ZeRO-3 and AutoEP EP 4 on 8 GPUs (bf16, Adam), giving 8 optimizer files of 36.19 GiB in total. Each version ran only `_consolidate_zero3_autoep_expert_states`, in its own process; peak resident memory is `ru_maxrss` (0.4 GiB of it is the imports). | | Peak host memory | Time | Output | |---|---|---|---| | master 3fe2d9508 | 109.20 GiB | 93.6 s | | | this change | 13.13 GiB | 203.6 s | all 36 universal files (fp32, exp_avg, exp_avg_sq of each expert weight) bitwise equal to master's | Master holds about 3 times the optimizer files at once; with this change the peak is about one optimizer file plus one parameter's tensors, and it does not grow with the number of ranks. The cost is time: the fragments are written to and read back from `<output>/tmp`, 2.2x slower here on a shared Lustre file system. For MiniMax-M3 and GLM-5.2 the old path cannot finish at all on a 2 TiB host. Hardware: 1 node of 8 NVIDIA H100 80GB (driver 595.112.07), torch 2.8.0+cu126. Probe script: `ucp_expert_memory_probe.py` (modes `save`, `convert`, `compare`). ## Tests On master 3fe2d9508 with this change, 1 node of 8 NVIDIA H100 80GB (driver 595.112.07), torch 2.8.0+cu126, pytest 8.3.5: | Test | Result | |---|---| | `tests/unit/checkpoint/test_autoep_affine_universal.py` (calls the function directly) and `tests/unit/v1/moe/test_autoep_checkpoint.py` (AutoEP ZeRO-3 save, convert, load at other layouts) | 69 cases: 67 passed, 2 skipped (ZeRO-1/2 universal-checkpoint cases that run only with `DEEPSPEED_RUN_H100_TESTS=1`; this change is on the ZeRO-3 path) | `pre-commit run --files` passes on the changed file. ## Duplicate check Searched open pull requests on 2026-11-13 ("ds_to_universal", "universal checkpoint AutoEP"). #9069 and #8973 change how ZeRO-3 loads universal checkpoints and partial-offload checkpoints, not `ds_to_universal.py`. #8848 (mine) also edits `_consolidate_zero3_autoep_expert_states` (keys by tensor-parallel and data-parallel rank); whichever of the two merges second needs a small rebase. ## Compatibility | Cell | Validated | |---|---| | ZeRO-3 + AutoEP checkpoint conversion, save / convert / load at other layouts | yes (tests) | | Dense (non-AutoEP) ZeRO-3 conversion | path unchanged | | ZeRO-1 / ZeRO-2 conversion | path unchanged | ## AI assistance This change, its tests and this description were prepared with AI assistance (Claude). --------- Signed-off-by: pengdurice <pengduhit@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Author
Parents
Loading