Universal Checkpointing: read AutoEP ZeRO-3 optimizer files one at a time (#8844)
# Universal Checkpointing: read AutoEP ZeRO-3 optimizer files one at a
time
## Problem
`ds_to_universal` converts a ZeRO-3 AutoEP checkpoint in two parts. The
dense parameters go through the usual
extract-and-merge path, which reads one optimizer file per worker and
writes fragments to `tmp/`. The routed experts
go through `_consolidate_zero3_autoep_expert_states`, which first loads
**every** rank's optimizer file into one dict
(`optim_states_by_rank`) and then keeps a copy of every expert fragment
in memory until all of them are merged.
Together those files hold every optimizer state of the model: fp32 (or
bf16) master weights plus both Adam moments.
For a large MoE model that is more than one host has:
| Model | Parameters | Optimizer state, fp32 master + fp32 Adam (12
B/param) | bf16 master + bf16 Adam (6 B/param) |
|---|---|---|---|
| MiniMax-M3 | 426.2B | 4.65 TiB | 2.33 TiB |
| GLM-5.2 | 743.4B | 8.11 TiB | 4.06 TiB |
The H200 nodes this was found on have 2 TiB of host memory, so the
conversion cannot finish for either model.
## Change
`_consolidate_zero3_autoep_expert_states` reads one rank's optimizer
file at a time:
- Each expert fragment it finds is written to
`<output>/tmp/autoep_expert_fragments/` (one file per parameter,
optimizer state, EP rank and partition rank).
- Every reference into the rank's optimizer state, including the loop
variables, is dropped before the next file is
read.
- The merge then loads, for one parameter and one state at a time, only
that parameter's fragments, and deletes each
fragment file once it is read, so the disk does not hold every fragment
and every universal file at the same time.
Peak host memory becomes one optimizer file plus one full expert tensor,
instead of all optimizer files.
The checks are unchanged: missing ranks, duplicate or missing
expert-data-parallel partitions, shape mismatches and
placement disagreements raise the same errors. The temporary fragments
live under `<output>/tmp`, which the converter
already uses for the dense fragments and removes at the end.
## Peak host memory
Measured with a probe on one checkpoint: a 4-layer mock MoE model (64
experts, hidden 1024, expert intermediate
4096: 3.22B expert parameters) trained one step and saved with ZeRO-3
and AutoEP EP 4 on 8 GPUs (bf16, Adam), giving
8 optimizer files of 36.19 GiB in total. Each version ran only
`_consolidate_zero3_autoep_expert_states`, in its own
process; peak resident memory is `ru_maxrss` (0.4 GiB of it is the
imports).
| | Peak host memory | Time | Output |
|---|---|---|---|
| master 3fe2d9508 | 109.20 GiB | 93.6 s | |
| this change | 13.13 GiB | 203.6 s | all 36 universal files (fp32,
exp_avg, exp_avg_sq of each expert weight) bitwise equal to master's |
Master holds about 3 times the optimizer files at once; with this change
the peak is about one optimizer file plus one
parameter's tensors, and it does not grow with the number of ranks. The
cost is time: the fragments are written to
and read back from `<output>/tmp`, 2.2x slower here on a shared Lustre
file system. For MiniMax-M3 and GLM-5.2 the
old path cannot finish at all on a 2 TiB host.
Hardware: 1 node of 8 NVIDIA H100 80GB (driver 595.112.07), torch
2.8.0+cu126. Probe script:
`ucp_expert_memory_probe.py` (modes `save`, `convert`, `compare`).
## Tests
On master 3fe2d9508 with this change, 1 node of 8 NVIDIA H100 80GB
(driver 595.112.07), torch 2.8.0+cu126, pytest 8.3.5:
| Test | Result |
|---|---|
| `tests/unit/checkpoint/test_autoep_affine_universal.py` (calls the
function directly) and `tests/unit/v1/moe/test_autoep_checkpoint.py`
(AutoEP ZeRO-3 save, convert, load at other layouts) | 69 cases: 67
passed, 2 skipped (ZeRO-1/2 universal-checkpoint cases that run only
with `DEEPSPEED_RUN_H100_TESTS=1`; this change is on the ZeRO-3 path) |
`pre-commit run --files` passes on the changed file.
## Duplicate check
Searched open pull requests on 2026-11-13 ("ds_to_universal", "universal
checkpoint AutoEP"). #9069 and #8973 change
how ZeRO-3 loads universal checkpoints and partial-offload checkpoints,
not `ds_to_universal.py`. #8848 (mine) also
edits `_consolidate_zero3_autoep_expert_states` (keys by tensor-parallel
and data-parallel rank); whichever of the two
merges second needs a small rebase.
## Compatibility
| Cell | Validated |
|---|---|
| ZeRO-3 + AutoEP checkpoint conversion, save / convert / load at other
layouts | yes (tests) |
| Dense (non-AutoEP) ZeRO-3 conversion | path unchanged |
| ZeRO-1 / ZeRO-2 conversion | path unchanged |
## AI assistance
This change, its tests and this description were prepared with AI
assistance (Claude).
---------
Signed-off-by: pengdurice <pengduhit@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>