DeepSpeed
Samyamr/full precision for ZeRO Stage2 and Stage3
#1004
Merged
Overview
Commits
21
Changes
View On GitHub

Samyamr/full precision for ZeRO Stage2 and Stage3 #1004

jeffra merged 21 commits into master from samyamr/full-precision-for-stage3
samyam
samyam Adding tf32 and fp32 support for ZeRO Stage 3
f4f28b94
samyam Changing to location of self.dtype assignment
08fc9e67
samyam Exhaustive setting of self.dtype
ed679c96
samyam Adding fp32 and tf32 support for ZeRO Stage 2
2eda1616
samyam fix loss scale value for static loss scale
5ce4d3bd
samyam Adding tf32 and fp32 support for ZeRO Stage 3
639d4875
samyam Changing to location of self.dtype assignment
e3b35342
samyam Exhaustive setting of self.dtype
bed15a01
samyam Adding fp32 and tf32 support for ZeRO Stage 2
bfe1e844
samyam fix loss scale value for static loss scale
7d762799
samyam adding documentation for dtype in zero init
0bf7efd9
samyam fixing conflict
6b250c20
samyam samyam requested a review from jeffra jeffra 4 years ago
samyam samyam requested a review from tjruwase tjruwase 4 years ago
samyam samyam requested a review from arashashari arashashari 4 years ago
samyam samyam requested a review from awan-10 awan-10 4 years ago
samyam samyam requested a review from cli99 cli99 4 years ago
samyam samyam requested a review from conglongli conglongli 4 years ago
samyam samyam requested a review from eltonzheng eltonzheng 4 years ago
samyam samyam requested a review from minjiaz minjiaz 4 years ago
samyam samyam requested a review from niumanar niumanar 4 years ago
samyam samyam requested a review from RezaYazdaniAminabadi RezaYazdaniAminabadi 4 years ago
samyam samyam requested a review from ShadenSmith ShadenSmith 4 years ago
tjruwase
tjruwase approved these changes on 2021-04-26
samyam
samyam commented on 2021-04-26
jeffra Merge branch 'master' into samyamr/full-precision-for-stage3
f8a4c8c8
stas00
tjruwase Merge branch 'master' into samyamr/full-precision-for-stage3
0d37b884
tjruwase Disable cpu-adam update_copy api for fp32
7260cb2d
tjruwase Disable gradient clipping in engine for ZeRO
b31088b8
tjruwase fp16 mode init required for ZeRO-3
cec23774
samyam samyam changed the title Samyamr/full precision for stage3 Samyamr/full precision for ZeRO Stage2 and Stage3 4 years ago
stas00
samyam Update engine.py
de76124f
tjruwase Formatting fix
bb46f588
jeffra bump DSE
ab35410a
jeffra
jeffra approved these changes on 2021-04-29
jeffra Merge branch 'master' into samyamr/full-precision-for-stage3
eb7901da
jeffra jeffra merged dad26428 into master 4 years ago
jeffra jeffra deleted the samyamr/full-precision-for-stage3 branch 4 years ago
stas00

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum