DeepSpeed
b5e0ba67 - Forward the per-head Muon tag at every muon_update call site (#8612)

Commit
1 day ago
Forward the per-head Muon tag at every muon_update call site (#8612) Per-head Muon (#8384) is silently ignored on three of the six paths that call `muon_update`, so the attention projections it tags are orthogonalized whole. `deepspeed.initialize` tags each attention projection with `muon_num_heads` and the call sites read it back off the parameter. Three do not: the padded branch of ZeRO 1/2 `get_flat_partition` (its unpadded twin does), `MuonWithAuxAdam.step`, and the new BF16_Optimizer Muon support. The padded branch runs when `zero_optimization.parameter_alignment` is on and a parameter's size is off the alignment, so whether the feature takes effect turns on an unrelated setting, and the update stays finite either way. Forward `num_heads` at all three. Test: `TORCHDYNAMO_DISABLE=1 DS_ACCELERATOR=cpu PYTHONPATH=.:tests python -m pytest tests/unit/runtime/zero/test_per_head_muon.py -q`. The new tests fail with the fix reverted and 80 pass after: a bf16 ZeRO 1 and 2 engine with `parameter_alignment: true` takes a step on the padded branch and checks `muon_update` gets `q_proj`'s head count; one steps `MuonWithAuxAdam` on a tagged projection and asserts the applied update is the per-head one rather than the whole-matrix one (0.32 apart here); and one names any call site that drops the tag. yapf 0.43.0 and flake8 5.0.4 pass on the changed files. Executed on Apple M2 Pro CPU (single-rank engine); no GPU or multi-rank run. --------- Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com> Co-authored-by: pengdurice <pengduhit@gmail.com>
Author
Parents
Loading