DeepSpeed
64616608 - Preserve convolution shapes and scaling in Muon updates (#8552)

Commit
1 day ago
Preserve convolution shapes and scaling in Muon updates (#8552) Muon crashes when updating convolution weights with four-dimensional gradients. The overflow guard combines a flattened update with the original four-dimensional gradient. The aspect-ratio correction also reads the kernel dimensions instead of the flattened matrix. Restore the original shape before the overflow guard and derive the scale from the matrix being orthogonalized. Test: `DS_ACCELERATOR=cpu PYTHONPATH=. python -m pytest tests/unit/v1/ops/muon/test_muon_conv.py -q` with torch.compile on. At the PR base all 10 fail with a shape mismatch; they pass after, including in a full-file run where a dynamic-shape recompile used to truncate the aspect-ratio scale. Changed files pass yapf and flake8 at the repo's pinned versions. Executed on Apple M2 Pro CPU; no CUDA or distributed training run. Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com> Co-authored-by: pengdurice <pengduhit@gmail.com>
Author
Parents
Loading