DeepSpeed
e75d1150 - Unmanaged gradient accumulation: ZeRO stage 3 support (#8217)

Commit
1 day ago
Unmanaged gradient accumulation: ZeRO stage 3 support (#8217) ## Summary - Extends unmanaged gradient accumulation (`managed_gradient_accumulation=false`) to **ZeRO stage 3** (non-offload). Stacked follow-up to #8203 (now merged). - Stage 3 partitions gradients into `__param_id_to_grad_partition` on every `backward()` (as in managed mode); `step()` only marks the boundary via `finalize_gradient_accumulation_boundary()`. - Since stage 3 is now permitted, also reject ZeRO **parameter** offload (previously unreachable behind the stage-3 guard). ZeRO optimizer-state offload, pipeline parallelism, DeepCompile, and Apex AMP remain unsupported pending follow-ups. ## Test plan Validated on a 2-GPU node: - [x] Full `-k Unmanaged` suite (29 passed), including `test_unmanaged_matches_managed[3]` and `test_unmanaged_varying_backward_count[3]` - [x] Stage-3 unmanaged feature subset (10 passed) - [x] Core ZeRO-3 regression (`test_zero_grad_clip`, `test_zero_leaf_module`, `test_zero_context`, `test_zero_tensor_fragment`): 158 passed, 62 skipped (expected) Docs (`config-json.md`, `training.rst`) updated; previewable on `rtd-staging`. Made with [Cursor](https://cursor.com) Signed-off-by: Olatunji Ruwase <tunji.ruwase@snowflake.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Parents
Loading