DeepSpeed
c687df79 - Harden ZeRO-1/2 offload gradient storage lifetime and stream ordering (#8632)

Commit
48 days ago
Harden ZeRO-1/2 offload gradient storage lifetime and stream ordering (#8632) ## Summary Harden gradient storage lifetime and stream ordering for ZeRO-1/2 with optimizer offload (`cpu` or `nvme`). These protections are always on for offload and have no configuration options. They address ownership and synchronization risks found while debugging NaNs under CPU offload. ## Changes - **Oversized-gradient ownership:** gradients larger than `reduce_bucket_size` are cloned into independent storage before reduction. Reduction and the D2H offload copy then read the clone. - **Producer readiness:** every bucket producer records an event, including oversized gradients. Reduction waits on all of them, with or without `overlap_comm`. - **Bucket reuse:** a contiguous IPG buffer records a reuse event when its consumer finishes, and later producers wait on it before writing into that buffer again. - **Stream lifetimes:** gradient tensors are `record_stream`'d on the streams that consume them. - **Offload copy ordering:** successive CPU-gradient writes are ordered across streams. Pending offload copies are waited on before CPU accumulator reloads, `step()`, `reset_cpu_buffers`, checkpoint loading (legacy and universal), and `destroy()`. - The existing `overlap_comm` barrier (compute stream waits on the reduction stream) is kept. The new events only add ordering on top of it. ## Scope - **Covered:** ZeRO-1 and ZeRO-2 with `offload_optimizer` set to `cpu` or `nvme`. Both stages use the same `reduce_independent_p_g_buckets_and_remove_grads` → `copy_grads_in_partition` D2H path, and offload already forces `contiguous_gradients`. Pipeline parallelism and DeepCompile are not excluded. - **Not covered:** ZenFlow, which overrides reduction and offload copies with its own ordering. A warning is logged when ZenFlow is combined with offload. - **Unchanged:** non-offloaded paths and ZeRO-3. Reduction thresholds, communication dtypes, and accumulation semantics are also unchanged. ## Cost Oversized gradients add one gradient-sized allocation and copy. Event tracking adds synchronization overhead and can extend storage lifetimes. ## Tests `tests/unit/v1/zero/test_zero2_gradient_safety.py`: - Unit tests for oversized-copy independence, producer and reuse waits, cross-stream CPU write ordering, the pageable-accumulator wait (default path, no accumulation flag), and waits before step, reset, and checkpoint load. - A constructor matrix covering stage, contiguous, and offload device (cpu, nvme, none). - Two-rank training against an independent full-batch reference: ZeRO-1/2, with and without offload, `overlap_comm`, and `contiguous_gradients=false`. - A CUDA stream-handoff test for oversized producers. Ran locally (macOS, CPU accelerator, 2 ranks): 35 passed, 2 skipped (the CUDA-only tests). A multi-GPU NCCL run of offload + overlap is still pending. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Jeff Rasley <jeff.rasley@snowflake.com> Signed-off-by: Jeff Rasley <jeffra45@gmail.com> Signed-off-by: tunji-ruwase_snow <tunji.ruwase@snowflake.com> Co-authored-by: Jeff Rasley <jeff.rasley@snowflake.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: tunji-ruwase_snow <tunji.ruwase@snowflake.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Author
Parents
Loading