DeepSpeed
Use expert gradients, not expert weights, in the FP16_Optimizer MoE grad norm
#8635
Merged

Use expert gradients, not expert weights, in the FP16_Optimizer MoE grad norm #8635

yh0903
yh0903 Fix MoE fp16 grad norm to use expert grads
43cb38c8
yh0903 yh0903 force pushed from ebe360e6 to 43cb38c8 57 days ago
yh0903 yh0903 marked this pull request as ready for review 56 days ago
yh0903 yh0903 requested a review from tjruwase tjruwase 56 days ago
yh0903 yh0903 requested a review from tohtana tohtana 56 days ago
yh0903 yh0903 requested a review from loadams loadams 56 days ago
pengdurice pengdurice added will-review
tohtana
tohtana approved these changes on 2026-09-29
tohtana tohtana enabled auto-merge 52 days ago
tohtana tohtana merged 1b60c549 into master 52 days ago
gpu-test-labeler gpu-test-labeler added skip-gpu-test

Login to write a write a comment.

Login via GitHub

Assignees
No one assigned
Labels
Milestone