DeepSpeed
Z3: optimizations for grad norm calculation and gradient clipping
#5504
Merged
Merged
Z3: optimizations for grad norm calculation and gradient clipping #5504
loadams
merged
25 commits into
deepspeedai:master
from
nelyahu:zero3_scaled_global
tjruwase
commented
on 2024-05-07
z3 scaled_global_grad_norm: repalce get_global_norm with torch.norm
43792bb7
Merge branch 'master' into zero3_scaled_global
cac04d9c
tjruwase
approved these changes
on 2024-05-20
Merge branch 'master' into zero3_scaled_global
cbb6b6a5
Merge branch 'master' into zero3_scaled_global
37b4cb77
fix grad norm calc in cpu offload and use torch.clip for grad clipping
5dd50c32
nelyahu
changed the title 
Merge branch 'master' into zero3_scaled_global
f918054a
Merge branch 'master' into zero3_scaled_global
ba9fd426
Merge branch 'master' into zero3_scaled_global
238ab343
Merge branch 'master' into zero3_scaled_global
6b6a834f
Merge branch 'master' into zero3_scaled_global
b14f920f
Merge branch 'master' into zero3_scaled_global
45e62d29
Merge branch 'master' into zero3_scaled_global
ffdb7f7d
adding gradient clipping to TestZeroPartialOffloadConfigSweep
d99524ba
Merge branch 'master' into zero3_scaled_global
e5d5d7c2
Merge branch 'master' into zero3_scaled_global
67d2e35a
Merge branch 'master' into zero3_scaled_global
214103df
Merge branch 'master' into zero3_scaled_global
6134da44
Merge branch 'master' into zero3_scaled_global
4b16d7ca
Merge branch 'master' into zero3_scaled_global
38bb8608
Merge branch 'master' into zero3_scaled_global
ec1aa8c0
Merge branch 'master' into zero3_scaled_global
95cd3a1f
Merge branch 'master' into zero3_scaled_global
26676f26
Merge branch 'master' into zero3_scaled_global
e4559624
Merge branch 'master' into zero3_scaled_global
3407e93c
Merge branch 'master' into zero3_scaled_global
cb51597f
Login to write a write a comment.
Login via GitHub