DeepSpeed
7d30f920 - feat(comm): one-shot P2P custom all-reduce for TP=2 inference (#8997)

Commit
4 days ago
feat(comm): one-shot P2P custom all-reduce for TP=2 inference (#8997) ## Summary Adds a latency-optimized one-shot all-reduce for P2P-capable CUDA pairs (TP=2), filling the existing `deepspeed.comm.inference_all_reduce` hook — the same slot the CPU backend fills with shm+AVX. **Split out of #8602**: this is a generic comm-layer optimization with zero dependency on segment-KI; any AutoTP TP=2 workload benefits, including plain `HF generate` whose decode steps all-reduce small hidden-state tensors through the TP-wrapped linears. ### How it works - `csrc/inference/custom_ar.cu`: one-shot P2P kernel over CUDA IPC buffers (2x4MB, sequence-numbered flags) - `deepspeed/comm/custom_ar.py`: topology gating — world==2, P2P capable, bounded-spin handshake probe; **symmetric verdict** via `all_reduce(MIN)` so a unidirectional fabric fault disables the fast path on both ranks (prevents the peer-spin hang) - `op_builder/custom_ar.py` (`DS_BUILD_CUSTOM_AR`): built through the standard op-builder path like every other op (wheel-prebuildable), initialized next to the shm op in `TorchBackend`, mirroring `build_shm_op` - `layers.py`: `RowParallel` routes no-grad small-message decode through `inference_all_reduce` (falls back to NCCL unchanged); inference mode uses reduce-scatter+all-gather - Fallbacks are safe and logged: non-CUDA, world≠2, no P2P, oversized (>4MB), or build failure → NCCL ## Not a duplicate Split from #8602 (segment-KI), which remains the umbrella PR. No other open PR implements a CUDA fast path for the `inference_all_reduce` hook (CPU shm path is `ShareMemCommBuilder`, untouched here). ## Tests `tests/unit/comm/test_custom_ar.py` — routing contract (4 tests, CPU-runnable): - not-ready before initialize; initialize() no-ops without a process group - payload gate is zero when not ready (no message can ever qualify) Run: `pytest tests/unit/comm/test_custom_ar.py tests/unit/comm/ -q` → 38 passed, 28 skipped on CPU (local, Python 3.11, CPU only). **On-device validation**: the kernel + handshake path was exercised on 2x GPU (AutoDL, see #8602 commit `4bcc86b82` verification) through the ZeRO-3→arena→TP2 rollout e2e. A dedicated 2-GPU parity test (custom_ar vs NCCL all-reduce) is planned as follow-up; reviewer may request hardware specifics from #8602's logs. ## Compatibility matrix Validated: CPU (comm unit suite), CUDA TP=2 (via #8602 e2e). Not validated: XPU/HPU/NPU (code routes through `get_accelerator()` and falls back; only `can_device_access_peer` is CUDA-specific, marked `#ignore-cuda`). ZeRO/offload unaffected (inference-path only). ## AI assistance This PR was prepared with AI assistance (code assembly, builder wiring, contract tests, review-hardening fixes — symmetric handshake verdict, `get_accelerator()` routing). The human submitter reviewed every line; the handshake-hang analysis and the split decision originated from human review discussion. --------- Signed-off-by: Ma, Guokai <guokai.ma@intel.com> Signed-off-by: Guokai Ma <guokai.ma@intel.com> Signed-off-by: Ma, Guokai <guokai.ma@gmail.com> Co-authored-by: pengdurice <pengduhit@gmail.com>
Author
Parents
Loading