Harden ZeRO-1/2 offload gradient storage lifetime and stream ordering (#8632)
## Summary
Harden gradient storage lifetime and stream ordering for ZeRO-1/2 with
optimizer offload (`cpu` or `nvme`). These protections are always on for
offload and have no configuration options. They address ownership and
synchronization risks found while debugging NaNs under CPU offload.
## Changes
- **Oversized-gradient ownership:** gradients larger than
`reduce_bucket_size` are cloned into independent storage before
reduction. Reduction and the D2H offload copy then read the clone.
- **Producer readiness:** every bucket producer records an event,
including oversized gradients. Reduction waits on all of them, with or
without `overlap_comm`.
- **Bucket reuse:** a contiguous IPG buffer records a reuse event when
its consumer finishes, and later producers wait on it before writing
into that buffer again.
- **Stream lifetimes:** gradient tensors are `record_stream`'d on the
streams that consume them.
- **Offload copy ordering:** successive CPU-gradient writes are ordered
across streams. Pending offload copies are waited on before CPU
accumulator reloads, `step()`, `reset_cpu_buffers`, checkpoint loading
(legacy and universal), and `destroy()`.
- The existing `overlap_comm` barrier (compute stream waits on the
reduction stream) is kept. The new events only add ordering on top of
it.
## Scope
- **Covered:** ZeRO-1 and ZeRO-2 with `offload_optimizer` set to `cpu`
or `nvme`. Both stages use the same
`reduce_independent_p_g_buckets_and_remove_grads` →
`copy_grads_in_partition` D2H path, and offload already forces
`contiguous_gradients`. Pipeline parallelism and DeepCompile are not
excluded.
- **Not covered:** ZenFlow, which overrides reduction and offload copies
with its own ordering. A warning is logged when ZenFlow is combined with
offload.
- **Unchanged:** non-offloaded paths and ZeRO-3. Reduction thresholds,
communication dtypes, and accumulation semantics are also unchanged.
## Cost
Oversized gradients add one gradient-sized allocation and copy. Event
tracking adds synchronization overhead and can extend storage lifetimes.
## Tests
`tests/unit/v1/zero/test_zero2_gradient_safety.py`:
- Unit tests for oversized-copy independence, producer and reuse waits,
cross-stream CPU write ordering, the pageable-accumulator wait (default
path, no accumulation flag), and waits before step, reset, and
checkpoint load.
- A constructor matrix covering stage, contiguous, and offload device
(cpu, nvme, none).
- Two-rank training against an independent full-batch reference:
ZeRO-1/2, with and without offload, `overlap_comm`, and
`contiguous_gradients=false`.
- A CUDA stream-handoff test for oversized producers.
Ran locally (macOS, CPU accelerator, 2 ranks): 35 passed, 2 skipped (the
CUDA-only tests). A multi-GPU NCCL run of offload + overlap is still
pending.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Jeff Rasley <jeff.rasley@snowflake.com>
Signed-off-by: Jeff Rasley <jeffra45@gmail.com>
Signed-off-by: tunji-ruwase_snow <tunji.ruwase@snowflake.com>
Co-authored-by: Jeff Rasley <jeff.rasley@snowflake.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: tunji-ruwase_snow <tunji.ruwase@snowflake.com>
Co-authored-by: Cursor <cursoragent@cursor.com>