ZeRO-3: fix stale and lost gradients of unused parameters (#8981)
## Summary
ZeRO-3 keeps each param's gradient partition across steps and overwrites
it only in micro-step 0. A param that
gets no gradient in a step (a skipped branch, an unused expert)
therefore keeps its last gradient, and the optimizer
applies it again; ZeRO-1/2 give it a zero gradient. With this PR, a
param's first gradient after `step()` or
`zero_grad()` overwrites its partition. At the accumulation boundary,
the partitions that got no gradient are
zeroed, and with optimizer offload, every partition the optimizer has
not received yet is copied to it.
## Fixes
Before → after, 1 rank on GH200 (base `2d80cedd2`):
1. A param without a gradient in a step was updated again with its last
gradient (SGD, largest weight change:
2.358e-02 → 0).
2. With gradient accumulation, a param first used after micro-step 0
added its gradient to the previous step's.
3. With optimizer offload, the gradient of a param used only in earlier
micro-steps never reached the optimizer
(norm 0 → 0.17340).
4. With fp16, an inf left in an unused param's partition made every
later step overflow (skipped steps: 5 → 1 of
5).
5. A param used by two reentrant checkpoint segments kept one of its two
gradients if they were in different
buckets
(2 ranks, norm 0.10933 → 0.27053).
---------
Signed-off-by: Xinyu Lian <lian7@illinois.edu>
Co-authored-by: Zhipeng Wang <zhipeng.rainbowserie@gmail.com>