DeepSpeed
f0166f96 - [AutoEP] Stage expert weight gradients across microbatches

Commit
1 day ago
[AutoEP] Stage expert weight gradients across microbatches With many experts or small microbatches each expert receives few rows per backward, and the grouped expert MLP's weight-gradient GEMMs write three full-size gradients for a handful of rows each. deferred_dw_flush_period stages the rows those GEMMs read in a per-expert buffer and runs the GEMMs once for several backward passes: every flush_period backward passes, early when an expert's cursor would overflow, and always in the backward that closes a gradient accumulation window. The flush returns its gradients through autograd, so gradient reduction and the optimizer see the same accumulated gradient. ZeRO-2's gradient hook now skips a parameter the backward delivered no gradient for. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: Zhipeng Wang <zhipeng.rainbowserie@gmail.com>
Author
Parents
Loading