DeepSpeed
fd8be816 - Exit spawned test workers with os._exit to avoid gloo abort at shutdown (#9100)

Commit
1 day ago
Exit spawned test workers with os._exit to avoid gloo abort at shutdown (#9100) Fixes #9089. ## Root cause A worker of `TestAutoTPCompileTiedEmbeddings::...[False]` occasionally aborts with `terminate called without an active exception` after the test body has passed. Native backtrace of the abort (CPU, gloo, torch 2.14, Python 3.12): ``` std::terminate _Unwind_ForcedUnwind pthread_exit PyEval_RestoreThread <- interpreter is finalizing libtorch_python ... c10::TensorImpl::decref_pyobject c10d::ProcessGroupGloo::runLoop ``` - Non-daemonic workers are started with the `spawn` method, so they run full interpreter finalization on exit. `fork`/`forkserver` children end with `os._exit`. - During finalization a gloo `runLoop` thread can still be releasing the last reference to a Python-owned tensor. Acquiring the GIL there makes CPython `pthread_exit` the thread, and the forced unwind through a `noexcept` frame calls `std::terminate`. - `destroy_process_group()` alone is not enough: with DeepCompile the compiled graph keeps the TP group alive (placeholder `meta['example_value']`), so the gloo threads are not joined. This reproduces without DeepSpeed: 2 spawned gloo ranks doing one `all_reduce` and returning without `destroy_process_group()` abort in 3 of 60 runs. ## Fix (test harness workaround) Spawned workers now exit through a small `mp.Process` subclass that calls `os._exit` after `_bootstrap`, matching the existing fork/forkserver behavior. No collective runs at teardown, so a rank whose test fails cannot leave the others blocked. ## Testing Local CPU repro (`DS_ACCELERATOR=cpu`, `DS_DISABLE_REUSE_DIST_ENV=1`, `pytest --forked -m sequential unit/compile/test_tp_compile.py::TestAutoTPCompileTiedEmbeddings`, repeated): - before: 9 aborted runs / 66 - after: 0 / 60 Harness semantics: a test failing on only one rank still reports `Worker 1 exited with code 1` with the assertion; `pytest.skip` inside workers is still propagated. Other non-daemonic tests on CPU (`unit/compile/test_tp_compile.py`, `unit/pipe/test_pipe_module.py`, `unit/comm/test_dist.py`, `unit/v1/compile/test_compile_zero.py`, `unit/v1/autotp/test_tp_plan_e2e.py`, `unit/v1/half_precision/test_keep_in_fp32_modules.py`): 139 passed, 62 skipped. --------- Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Author
Parents
Loading