PR #275 ZeRO-2: Handle gradients of empty partitions

Load non-DeepSpeed checkpoints into ZeRO optimizer

tjruwase committed 5 years ago

Handle parameters smaller than DP

tjruwase committed 5 years ago

Formatting fixes

tjruwase committed 5 years ago

Handle empty partitions

tjruwase committed 5 years ago

Fix perf bug

tjruwase committed 5 years ago

Merge with master

tjruwase committed 5 years ago

Merge branch 'master' into olruwase/zero2_empty_partition

tjruwase committed 5 years ago

Merge branch 'master' into olruwase/zero2_empty_partition

jeffra committed 5 years ago

DeepSpeed ZeRO-2: Handle gradients of empty partitions #275 Merged