Restore fp32 masters for ZeRO-3 and the ZeRO-1 BF16 optimizer #8892
sfc-gh-truwase
wants to merge
9 commits into
deepspeedai:master
from
sfc-gh-truwase:load-fp32-masters-stage3-bf16
Keep fp32 masters on ZeRO-1 and ZeRO-2 when a resume skips optimizer …
f9958d4d
Restore fp32 masters for ZeRO-3 and the ZeRO-1 BF16 optimizer.
bb31f403
Merge master into the ZeRO-1/2 fp32 master branch.
b3728af3
Clarify FP32 checkpoint loading paths
e489f80b
Keep saved fp32 masters on every full ZeRO load.
a7ce02d9
Fail a universal load that is missing its fp32 master.
2f56cafc
Skip DistributedFixture tests when the world size is unavailable.
894a2b72
Merge the ZeRO-1/2 fp32 master branch into the ZeRO-3 and BF16 follow…
e3b65b61
Test ZeRO-3 and BF16 fp32 masters when optimizer state is skipped.
b190a3f7
Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub