DeepSpeed
daf175b1 - ZeRO-3: fix stale and lost gradients of unused parameters (#8981)

Commit
1 day ago
ZeRO-3: fix stale and lost gradients of unused parameters (#8981) ## Summary ZeRO-3 keeps each param's gradient partition across steps and overwrites it only in micro-step 0. A param that gets no gradient in a step (a skipped branch, an unused expert) therefore keeps its last gradient, and the optimizer applies it again; ZeRO-1/2 give it a zero gradient. With this PR, a param's first gradient after `step()` or `zero_grad()` overwrites its partition. At the accumulation boundary, the partitions that got no gradient are zeroed, and with optimizer offload, every partition the optimizer has not received yet is copied to it. ## Fixes Before → after, 1 rank on GH200 (base `2d80cedd2`): 1. A param without a gradient in a step was updated again with its last gradient (SGD, largest weight change: 2.358e-02 → 0). 2. With gradient accumulation, a param first used after micro-step 0 added its gradient to the previous step's. 3. With optimizer offload, the gradient of a param used only in earlier micro-steps never reached the optimizer (norm 0 → 0.17340). 4. With fp16, an inf left in an unused param's partition made every later step overflow (skipped steps: 5 → 1 of 5). 5. A param used by two reentrant checkpoint segments kept one of its two gradients if they were in different buckets (2 ranks, norm 0.10933 → 0.27053). --------- Signed-off-by: Xinyu Lian <lian7@illinois.edu> Co-authored-by: Zhipeng Wang <zhipeng.rainbowserie@gmail.com>
Author
Parents
Loading