DeepSpeed
ZeRO 3 Offload
#834
Merged
Overview
Commits
17
Changes
View On GitHub

ZeRO 3 Offload #834

samyam merged 17 commits into master from staging-zero3-release
samyam
jeffra Squash stage3 v1 (#146)
d53ccaff
jeffra Fix correctness bug (#147)
4179ddae
formatting fix (#150)
3a2d5cd0
stage3 bugfix (API) update and simplified FP16 Z3 tests (#151)
fee24912
ZeRO-3 detach and race condition bugfixes (#149)
fe21d210
tjruwase Fix optimizer state_dict KeyError (#148)
4b2838f2
jeffra fix for smaller SGS sizes, ensures each grad is backed by unique tens…
2e9025f2
samyam Simplifying the logic for getting averaged gradients (#153)
b447a2ac
jeffra skip for now
fb0d4fb7
Z3 Docs redux (#154)
8013615a
removing some TODOs and commented code (#155)
591ca5ec
New Z3 defaults (#156)
d21a838b
jeffra formatting
7dec8895
samyam samyam requested a review from arashashari arashashari 5 years ago
samyam samyam requested a review from awan-10 awan-10 5 years ago
samyam samyam requested a review from cli99 cli99 5 years ago
samyam samyam requested a review from conglongli conglongli 5 years ago
samyam samyam requested a review from eltonzheng eltonzheng 5 years ago
samyam samyam requested a review from jeffra jeffra 5 years ago
samyam samyam requested a review from minjiaz minjiaz 5 years ago
samyam samyam requested a review from niumanar niumanar 5 years ago
samyam samyam requested a review from RezaYazdaniAminabadi RezaYazdaniAminabadi 5 years ago
samyam samyam requested a review from ShadenSmith ShadenSmith 5 years ago
samyam samyam requested a review from tjruwase tjruwase 5 years ago
megatron external params
0abd60f2
jeffra Merge branch 'master' into staging-zero3-release
0f22ae1f
Merge branch 'staging-zero3-release' of github.com:microsoft/DeepSpee…
9efdb6bc
jeffra Merge branch 'master' into staging-zero3-release
de7e45d1
jeffra
jeffra approved these changes on 2021-03-08
ShadenSmith
ShadenSmith approved these changes on 2021-03-08
samyam samyam merged 599258f9 into master 5 years ago
RezaYazdaniAminabadi
RezaYazdaniAminabadi
mrwyattii mrwyattii deleted the staging-zero3-release branch 2 years ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum