DeepSpeed
Count each expert once in the fp32 gradient-clipping norm
#8478

Open

Count each expert once in the fp32 gradient-clipping norm #8478

alanhuangyoo wants to merge 3 commits into deepspeedai:master from alanhuangyoo:fix/fp32-clip-moe-norm

Count each expert once in the fp32 gradient-clipping norm

875e8fe8

alanhuangyoo requested a review from

tjruwase 7 days ago

alanhuangyoo requested a review from

tohtana 7 days ago

alanhuangyoo requested a review from

loadams 7 days ago

alanhuangyoo marked this pull request as draft 7 days ago

Decide the expert path from ownership, not from this step's gradients

338ed514

alanhuangyoo marked this pull request as ready for review 7 days ago

ebarkhordar commented on 2026-09-13

Vote on the expert path across stages, and not at all without MoE

1a761e8d

Reviewers

ebarkhordar

tjruwase

tohtana

loadams

Assignees

No one assigned

Labels

None yet

Milestone

No milestone