PR #275 ZeRO-2: Handle gradients of empty partitions

ZeRO-2: Handle gradients of empty partitions #275

tjruwase merged 8 commits into master from olruwase/zero2_empty_partition

Load non-DeepSpeed checkpoints into ZeRO optimizer

c023cc6a

Handle parameters smaller than DP

69d17ab7

Formatting fixes

f2c74c54

Handle empty partitions

a1ff4785

Fix perf bug

df91f59d

Merge with master

93b13ebe

tjruwase requested a review from

jeffra 5 years ago

tjruwase requested a review from

samyam 5 years ago

tjruwase requested a review from

eltonzheng 5 years ago

Merge branch 'master' into olruwase/zero2_empty_partition

222606c8

Merge branch 'master' into olruwase/zero2_empty_partition

3e87d29c

jeffra approved these changes on 2020-07-02

tjruwase merged 4a3234e0 into master 5 years ago

jeffra deleted the olruwase/zero2_empty_partition branch 4 years ago

Reviewers

jeffra

samyam

eltonzheng

Assignees

No one assigned

Labels

None yet

Milestone

No milestone