DeepSpeed
Fix unbalanced gradients bug in ZeRO-2 gradient accumulation
#545
Merged
Overview
Commits
4
Changes
View On GitHub

Fix unbalanced gradients bug in ZeRO-2 gradient accumulation #545

tjruwase merged 4 commits into master from olruwase/zero2_grad_accum_bug
tjruwase
tjruwase Use zero-tensors for missing gradients to avoid size mismatch
0bb0cc80
tjruwase Unit test for unbalanced gradients in ZeRO
e1ff8bab
tjruwase tjruwase requested a review from jeffra jeffra 5 years ago
tjruwase tjruwase requested a review from samyam samyam 5 years ago
tjruwase tjruwase requested a review from eltonzheng eltonzheng 5 years ago
tjruwase tjruwase requested a review from arashashari arashashari 5 years ago
tjruwase tjruwase requested a review from awan-10 awan-10 5 years ago
tjruwase tjruwase requested a review from cli99 cli99 5 years ago
tjruwase tjruwase requested a review from conglongli conglongli 5 years ago
tjruwase tjruwase requested a review from minjiaz minjiaz 5 years ago
tjruwase tjruwase requested a review from niumanar niumanar 5 years ago
tjruwase tjruwase requested a review from RezaYazdaniAminabadi RezaYazdaniAminabadi 5 years ago
tjruwase tjruwase requested a review from ShadenSmith ShadenSmith 5 years ago
tjruwase Merge branch 'master' of github.com:microsoft/DeepSpeed into olruwase…
6a7829a1
eltonzheng
eltonzheng approved these changes on 2020-11-20
jeffra
jeffra approved these changes on 2020-11-20
tjruwase Formatting fixes
ce5d9074
tjruwase tjruwase merged 0178e6cc into master 5 years ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum