DeepSpeed
f51ffa97 - [AutoEP] Run staged expert weight gradients with ZeRO-3

Commit
3 hours ago
[AutoEP] Run staged expert weight gradients with ZeRO-3 ZeRO-3's gradient hook now skips a parameter the backward delivered no gradient for, and the grouped-GEMM expert paths copy a weight that ZeRO-3 gathered to an address torch._grouped_mm does not accept. The deferred expert function saves detached aliases of the weights, since ZeRO-3 swaps a parameter's data after the forward. Correct accumulation across windows additionally needs a parameter's first gradient of a window to overwrite its ZeRO-3 gradient partition, which #8981 provides. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Author
Parents
Loading