ZeRO Gradient Accumulation Dtype. #2847
jomayeri
merged
60 commits into
master
from
jomayeri/new-zero-accum
Adding attributes for grad accum dtype.
faa8e16c
accumulating reduction grads in stage 2 mode 2
f7c35615
missing colon
04cd6a74
tracking reduc grad move
4b878848
Merge branch 'master' into jomayeri/new-zero-accum
8570eca7
Merge branch 'master' into jomayeri/new-zero-accum
b48629c5
Correct hooks.
33f9d203
Merge remote-tracking branch 'refs/remotes/origin/jomayeri/new-zero-a…
543a7d3a
Merge branch 'master' into jomayeri/new-zero-accum
9022ce37
stas00
commented
on 2023-02-21
Merge branch 'master' into jomayeri/new-zero-accum
f137bfa4
Name change updates.
31a16447
Merge remote-tracking branch 'refs/remotes/origin/jomayeri/new-zero-a…
bb9b6a69
Using grad_accum in cpu offload functions.
a7d6b60a
Merge branch 'master' into jomayeri/new-zero-accum
9c6c4d36
Merge branch 'master' into jomayeri/new-zero-accum
f6f36724
tjruwase
commented
on 2023-02-24
tjruwase
commented
on 2023-02-24
Merge branch 'master' into jomayeri/new-zero-accum
63e6cc4f
Merge branch 'master' into jomayeri/new-zero-accum
2ca8199c
Addressing comments: putting bf opt back, removing hooks
ef80b4c4
Fixing missing pointer to grad accum.
4e0f920b
Merge branch 'master' into jomayeri/new-zero-accum
a9d86e40
Merge branch 'master' into jomayeri/new-zero-accum
fa406b14
tjruwase
commented
on 2023-03-01
Merge branch 'master' into jomayeri/new-zero-accum
c3fff11b
tjruwase
commented
on 2023-03-01
Renaming functions.
09b2eea9
Merge branch 'master' into jomayeri/new-zero-accum
aabe5a5e
Merge remote-tracking branch 'refs/remotes/origin/jomayeri/new-zero-a…
73860770
More function renames.
d65907a0
Merge branch 'master' into jomayeri/new-zero-accum
96d37af2
Merge branch 'master' into jomayeri/new-zero-accum
7cf1ddbf
Adding reduction dtype.
e0eb4e63
Merge branch 'master' into jomayeri/new-zero-accum
814c20a3
tjruwase
commented
on 2023-03-22
tjruwase
commented
on 2023-03-22
tjruwase
commented
on 2023-03-22
tjruwase
commented
on 2023-03-22
tjruwase
commented
on 2023-03-22
tjruwase
commented
on 2023-03-22
Merge branch 'master' into jomayeri/new-zero-accum
5c9d4f58
Merge branch 'master' into jomayeri/new-zero-accum
0e4271ad
Merge branch 'master' into jomayeri/new-zero-accum
e783894b
Merge branch 'master' into jomayeri/new-zero-accum
830180eb
Merge branch 'jomayeri/new-zero-accum' of github.com:microsoft/DeepSp…
cd30ca2c
Merge branch 'master' into jomayeri/new-zero-accum
9c7074d7
Merge branch 'master' into jomayeri/new-zero-accum
cce6cf16
updating for offload
d741446c
Merge branch 'master' into jomayeri/new-zero-accum
c5387fee
Merge branch 'master' into jomayeri/new-zero-accum
ac0a73c8
Merge branch 'master' into jomayeri/new-zero-accum
3a2280f5
Merge branch 'master' into jomayeri/new-zero-accum
8c3bfe1d
Adding functionality for stage 3.
b2c5f5ce
jomayeri
changed the title
Merge branch 'master' into jomayeri/new-zero-accum
25d42962
Adding s3 test support.
49623cad
Merge branch 'master' into jomayeri/new-zero-accum
15ebfda1
Add to MiCS optimizer.
1d31297c
zero++ tutorial PR (#3783)
df1859d6
Merge branch 'master' into jomayeri/new-zero-accum
22fa5591
Merge branch 'master' into jomayeri/new-zero-accum
6cd7a222
Merge branch 'master' into jomayeri/new-zero-accum
993ddc1e
Merge branch 'master' into jomayeri/new-zero-accum
df29a1e2
Merge branch 'master' into jomayeri/new-zero-accum
94ab3b87
Merge branch 'master' into jomayeri/new-zero-accum
6cad2a25
Merge branch 'master' into jomayeri/new-zero-accum
62a82368
Removing need to grad_reduc attribute.
26cefc57
Merge branch 'master' into jomayeri/new-zero-accum
e2129f0a
Merge branch 'master' into jomayeri/new-zero-accum
88f6aa2e
tjruwase
approved these changes
on 2023-07-19
Offload correctness.
0fab8df6
Merge branch 'master' into jomayeri/new-zero-accum
8492da7d
Login to write a write a comment.
Login via GitHub