[AutoEP] Stage expert weight gradients across microbatches
With many experts or small microbatches each expert receives few rows per
backward, and the grouped expert MLP's weight-gradient GEMMs write three
full-size gradients for a handful of rows each. deferred_dw_flush_period
stages the rows those GEMMs read in a per-expert buffer and runs the GEMMs
once for several backward passes: every flush_period backward passes, early
when an expert's cursor would overflow, and always in the backward that
closes a gradient accumulation window.
The flush returns its gradients through autograd, so gradient reduction and
the optimizer see the same accumulated gradient. ZeRO-2's gradient hook now
skips a parameter the backward delivered no gradient for.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Zhipeng Wang <zhipeng.rainbowserie@gmail.com>