DeepSpeed
7a31fe66 - Add managed_gradient_accumulation for ZeRO stage 0/1 (#8184)

Commit
15 days ago
Add managed_gradient_accumulation for ZeRO stage 0/1 (#8184) When managed_gradient_accumulation=false, disable micro-step tracking and treat each engine.step() as the accumulation boundary: reduce locally accumulated grads then apply the optimizer update. Stage 2/3 and pipeline remain unsupported in this change. Part fix for #8183 --------- Signed-off-by: Olatunji Ruwase <tunji.ruwase@snowflake.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Parents
Loading