DeepSpeed
Use expert gradients, not expert weights, in the FP16_Optimizer MoE grad norm
#8635
Merged

Loading