fix test_multiple_request_boundaries_before_step (#9028)
Fixes: https://github.com/deepspeedai/DeepSpeed/issues/9025
**Align the request-boundary regression with the distributed test
harness's process lifecycle.** Replace `torch.multiprocessing.spawn`
with `torch.multiprocessing.start_processes(...,
start_method="forkserver")` in
`test_multiple_request_boundaries_before_step`. The existing
`DistributedTest` harness also uses forkserver by default.
Issue #9025 reports intermittent CPU/Gloo child-process `SIGABRT` with
`terminate called without an active exception`. The numerical regression
itself is unchanged: two ranks independently initialize and tear down an
engine and process group, and every gradient and parameter-update
assertion still runs. No retries, ignored failures, tolerance changes,
or production-code changes are introduced.
## Rationale And Limit
**This is a lifecycle mitigation, not a confirmed native root-cause
fix.** Spawned workers exit through normal interpreter shutdown, which
runs native global destructors after explicit engine and distributed
cleanup. CPython's forkserver workers exit through `os._exit` after the
process bootstrap returns. The change avoids that additional
interpreter-global shutdown phase while retaining the test's explicit
cleanup and the launcher's child-failure propagation.
The original SIGABRT did not reproduce locally: 144 CPU cases passed on
PyTorch 2.12.1, 32 CPU cases passed on PyTorch 2.9.1, and the original
16-case matrix passed both under `--forked -n 4` on CPU and on two
H200s. These passing baselines prevent claiming a demonstrated
before/after fix. Repeated validation in the reporting CI environment is
still needed before declaring the intermittent abort eliminated.
## Validation
**All matrix runs use two ranks and exercise all 16 combinations.**
Coverage is ZeRO-1/2, device optimizer/CPU optimizer offload, BF16/FP32
gradient accumulation with BF16 model parameters, and
plain/reentrant-checkpointed forwards. The optimizer is SGD, matching
the existing test.
| Changed-launcher validation | Result |
| --- | --- |
| CPU/Gloo, PyTorch 2.12.1+cpu, five stress rounds | 80 passed |
| CPU/Gloo, PyTorch 2.12.1+cpu, `--forked -n 4` | 16 passed in 20.94 s |
| CPU/Gloo, PyTorch 2.12.1+cpu, serial `--forked` | 16 passed in 66.95 s
|
| CPU/Gloo, PyTorch 2.9.1+cu130 | 16 passed in 71.57 s |
| CUDA/NCCL, PyTorch 2.9.1+cu130, two H200s | 16 passed in 127.56 s |
**GPU validation ran on `stas-dev-4-0` with two NVIDIA H200s.** Each GPU
reports 143771 MiB; the NVIDIA driver is 580.159.03 and CUDA is 13.0.
The existing `dev` conda interpreter was used. The CPU runs also used
that interpreter, with the CPU-only torch wheel exposed from a temporary
directory where required. No conda or virtual environment was created or
modified.
**Worker failures still fail pytest.** A scratch copy deliberately added
1 to the expected gradient. The worker raised `AssertionError`, the
parent received `ProcessRaisedException`, and pytest exited with code 1.
The repository test's expected gradients were not altered.
Commands, from the DeepSpeed checkout using
`/home/yak/miniconda3/envs/dev/bin/python`:
```bash
TARGET=tests/unit/v1/zero/test_zero2_offload_multi_backward.py::test_multiple_request_boundaries_before_step
PY=/home/yak/miniconda3/envs/dev/bin/python
DS_ACCELERATOR=cpu OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 \
PYTHONPATH=/tmp/ds9025-torch212:tests:. \
"$PY" -m pytest -sv --color=no "$TARGET"
DS_ACCELERATOR=cpu OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 \
PYTHONPATH=/tmp/ds9025-ci-plugin:/tmp/ds9025-torch212:tests:. \
"$PY" -m pytest -sv --forked -n 4 --color=no --torch_ver=2.12.1+cpu "$TARGET"
# Executed on stas-dev-4-0.
CUDA_VISIBLE_DEVICES=0,1 DS_ACCELERATOR=cuda \
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 \
"$PY" -m pytest -sv --color=no "$TARGET"
```
The temporary paths expose the CPU-only torch wheel and the missing
pytest-forked dependency without replacing the existing pytest 8.4.1
installation. The CPU matrix was repeated five times. All applicable
pre-commit hooks passed for the changed test, including yapf, flake8,
license, and accelerator-import checks. `git diff --check` passed. These
timings are test durations, not training-throughput benchmarks.
## Compatibility And Submission
**No runtime, configuration, checkpoint, or public API behavior
changes.** The engine, ZeRO-1/2 and ZeRO-3 destruction paths, BF16
optimizer destruction, and distributed harness were inspected; no
sibling production changes are proposed. The full CPU suite, other
accelerators, ZeRO-3, NVMe offload, FP16, and additional parallelism
modes were not validated. Production convergence, throughput, and
peak-memory measurements are not applicable to this test-only launcher
change.
Public GitHub API searches for issue 9025, request-boundary tests, and
SIGABRT found no matching open PR during the November 6, 2026
investigation. This proposal does not duplicate the earlier
request-boundary gradient fix: it changes only test-process lifetime.
AI assistance was used. A human submitter must review the changed lines
and validation limitations before submitting and providing any required
sign-off. No commit, push, or PR creation was performed by this task.
Signed-off-by: Stas Bekman <stas@stason.org>
Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>
