DeepSpeed
Fix expert grad scaling problem with ZeRO optimizer
#6546

Merged

Fix expert grad scaling problem with ZeRO optimizer #6546

tohtana merged 7 commits into deepspeedai:master from wyooyw:fix_expert_weight_grad_with_zero

Fix Expert Grad Scale Problem With Zero Optimizer

607d8c9d

wyooyw requested a review from

tjruwase 1 year ago

wyooyw requested a review from

loadams 1 year ago

wyooyw changed the title ~~Fix Expert Grad Scaling Problem With Zero Optimizer~~ Fix expert grad scaling problem with ZeRO optimizer 1 year ago

tjruwase removed review request from

loadams 1 year ago

tjruwase requested a review from

tohtana 1 year ago

tohtana commented on 2024-09-17

remove useless code

5a44f8c0

ranzhejiang commented on 2024-09-18

remove useless comments

b1231c48

wyooyw force pushed from 6e1e90c1 to b1231c48 1 year ago

Merge branch 'master' into fix_expert_weight_grad_with_zero

14d002df

Merge branch 'master' into fix_expert_weight_grad_with_zero

76dda2a4

Merge branch 'master' into fix_expert_weight_grad_with_zero

d0de160c

Merge branch 'master' into fix_expert_weight_grad_with_zero

28b2aff4

tohtana enabled auto-merge 1 year ago

tohtana approved these changes on 2024-10-14

tohtana merged b647fb24 into master 1 year ago

Reviewers

tohtana

ranzhejiang

tjruwase

Assignees

No one assigned

Labels

None yet

Milestone

No milestone