DeepSpeed
25eaac4f - ZeRO-3 partial offload: fix LR schedules and checkpoints of the device sub-groups (#8973)

Commit
1 day ago
ZeRO-3 partial offload: fix LR schedules and checkpoints of the device sub-groups (#8973) ## Summary With `offload_optimizer.ratio` < 1, ZeRO-3 keeps some sub-groups on the device and steps them with a separate torch optimizer (`backup_optimizer`). That optimizer has several bugs that change the training result without an error. SuperOffload uses the same optimizer for its device sub-groups; the rest of SuperOffload is fixed in #8972. ## Fixes 1. The device sub-groups ignored LR schedules and `set_lr()`, and changes of `betas`, `eps` and `weight_decay`. Ina first step at lr 0, the weights moved by 3.0e-3. The backup optimizer now copies these values before each step. 2. Checkpoints had no Adam state for the device sub-groups (a resume was 4.9e-3 off after one step). It is now saved with the state of the CPU sub-groups: a resume is bit-identical, and a checkpoint loads with another `ratio`. 3. After a checkpoint load, every device step also updated the CPU sub-groups of param groups 1 and up, because their dicts were shared with the main optimizer (3.0e-3 extra per step at lr 3e-3). Each param group of the backup optimizer now has its own dict. 4. `safe_get_full_optimizer_state` and `safe_set_full_optimizer_state` raised `KeyError` for the device sub-groups, and `ds_to_universal` raised `KeyError: 0`. Both work now. 5. With `adam_w_mode: false`, the device sub-groups used decoupled weight decay (2.4e-2 off after 4 steps); they now use `torch.optim.Adam` (L2). `bias_correction: false` with `ratio` < 1 now raises a `ValueError`. --------- Signed-off-by: Xinyu Lian <lian7@illinois.edu>
Author
Parents
Loading