DeepSpeed
1b60c549 - Use expert gradients, not expert weights, in the FP16_Optimizer MoE grad norm (#8635)

Commit
52 days ago
Use expert gradients, not expert weights, in the FP16_Optimizer MoE grad norm (#8635) `FP16_Optimizer.step` builds two gradient-norm inputs, and the two consumers expect different things: - the non-expert list goes to `get_flattened_grad_norm`, which filters on `p.grad` and reads `p.grad.data`, so it expects **parameters**; - the expert list goes to `get_norm_with_moe_layers` → `get_global_norm_of_tensors`, which reads each tensor's own data, and whose docstring asks for **gradient tensors**. Both lists were filled with the FP32 master parameter `self.fp32_groups_flat[i]`. That is correct for the first consumer and wrong for the second, so the expert contribution to the global norm was the expert **weight** norm rather than the expert gradient norm. With the default `gradient_clipping=1.0`, that inflates the norm and shrinks the clip scale applied to every gradient, including non-expert ones. The fix passes `grads_groups_flat[i]` to the expert list. `BF16_Optimizer` (`self.fp32_groups_gradients[i][j]`) and `FP16_UnfusedOptimizer` (`split_params_grads_into_shared_and_expert_params`) already take the expert norm from gradients and are not affected. ### Test `TestMoEFP16GradientNorm::test_expert_weight_values_do_not_affect_gradient_norm` trains a small MoE model with fp16 and ZeRO stage 0, which selects `FP16_Optimizer`. It sets every gradient to the same fixed value, scales only the expert weights by 100, and requires the norm reported by `engine.get_global_grad_norm()` to be exactly unchanged. It also checks that the norm is above the default clipping threshold of 1.0, so that clipping acts on it. H100 80GB, torch 2.10 (CUDA 13.0): | Source | Result | |---|---| | test only, unmodified master | fails: `Scalars are not equal!` | | test + fix | passes | These runs predate the rebase onto current master, which does not touch either file. The test needs fp16, so CPU runners skip it. Signed-off-by: yh0903 <helloyu0903@gmail.com>
Author
Parents
Loading