DeepSpeed
40a9f40b - ZeRO-1/2: fix offload_states() keeping the optimizer state on the device (#9016)

Commit
4 hours ago
ZeRO-1/2: fix offload_states() keeping the optimizer state on the device (#9016) ## Summary ZeRO-1/2 links each parameter's optimizer-state fragment (`_hp_mapping.optim_fragment`, which `safe_get_full_optimizer_state()` and `safe_set_full_optimizer_state()` use) once, after the first step, as `narrow()` views of the optimizer state. `offload_states()` and `reload_states()` replace the state tensors but never link them again: 1. `offload_states(include=[optim_states])` frees none of the device copy of the state, because the views keep it alive. `reload_states()` then adds a second device copy, and the fragments keep pointing at the old one, so the getters return values from before the offload. 2. Reloading `hp_params` or `lp_params`, also with the default `include=None`, rebuilds the fragments through `_link_all_hp_params()` without the optimizer state, so from then on the getters raise `TypeError: argument of type 'NoneType' is not iterable`. This PR drops the fragments' views before the state is offloaded and links the state into the fragments again at the end of `reload_states()`. Before the first step there is no state yet, and `step()` links it as before. ## Fixes Before → after, 2 GH200, ZeRO-1 and ZeRO-2, bf16, FusedAdam, 33.6M parameters (128 MiB of Adam state per rank), base `a38957c17`: 1. `offload_states(include=[optim_states])`: 0 → 128 MiB freed; device memory after `reload_states()` 448.1 → 320.1 MiB (as before the offload); one step later the fragments were up to 2.5e-4 off the optimizer state → equal. 2. `include=[hp_params]`, `[lp_params]` or all states: `TypeError` from the getters after `reload_states()` → they return the current state. --------- Signed-off-by: Xinyu Lian <lian7@illinois.edu> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Author
Parents
Loading