DeepSpeed
Fix unbalanced gradients bug in ZeRO-2 gradient accumulation
#545
Merged

Fix unbalanced gradients bug in ZeRO-2 gradient accumulation #545

tjruwase merged 4 commits into master from olruwase/zero2_grad_accum_bug
tjruwase
tjruwase Use zero-tensors for missing gradients to avoid size mismatch
0bb0cc80
tjruwase Unit test for unbalanced gradients in ZeRO
e1ff8bab
tjruwase tjruwase requested a review from jeffra jeffra 6 years ago
tjruwase tjruwase requested a review from samyam samyam 6 years ago
tjruwase tjruwase requested a review from eltonzheng eltonzheng 6 years ago
tjruwase tjruwase requested a review from arashashari arashashari 6 years ago
tjruwase tjruwase requested a review from awan-10 awan-10 6 years ago
tjruwase tjruwase requested a review from cli99 cli99 6 years ago
tjruwase tjruwase requested a review from conglongli conglongli 6 years ago
tjruwase tjruwase requested a review from minjiaz minjiaz 6 years ago
tjruwase tjruwase requested a review from niumanar niumanar 6 years ago
tjruwase tjruwase requested a review from RezaYazdaniAminabadi RezaYazdaniAminabadi 6 years ago
tjruwase tjruwase requested a review from ShadenSmith ShadenSmith 6 years ago
tjruwase Merge branch 'master' of github.com:microsoft/DeepSpeed into olruwase…
6a7829a1
eltonzheng
eltonzheng approved these changes on 2020-11-20
jeffra
jeffra approved these changes on 2020-11-20
tjruwase Formatting fixes
ce5d9074
tjruwase tjruwase merged 0178e6cc into master 6 years ago

Login to write a write a comment.

Login via GitHub

Assignees
No one assigned
Labels
Milestone