DeepSpeed
Count each expert once in the fp32 gradient-clipping norm
#8478

Open

Count each expert once in the fp32 gradient-clipping norm #8478

alanhuangyoo wants to merge 2 commits into deepspeedai:master from alanhuangyoo:fix/fp32-clip-moe-norm

Count each expert once in the fp32 gradient-clipping norm

875e8fe8

alanhuangyoo requested a review from

tjruwase 16 hours ago

alanhuangyoo requested a review from

tohtana 16 hours ago

alanhuangyoo requested a review from

loadams 16 hours ago

alanhuangyoo marked this pull request as draft 6 hours ago

Decide the expert path from ownership, not from this step's gradients

338ed514

alanhuangyoo marked this pull request as ready for review 6 hours ago

Reviewers

tjruwase

tohtana

loadams

Assignees

No one assigned

Labels

None yet

Milestone

No milestone