Exit spawned test workers with os._exit to avoid gloo abort at shutdown (#9100)
Fixes #9089.
## Root cause
A worker of `TestAutoTPCompileTiedEmbeddings::...[False]` occasionally
aborts with `terminate called without an active exception` after the
test body has passed.
Native backtrace of the abort (CPU, gloo, torch 2.14, Python 3.12):
```
std::terminate
_Unwind_ForcedUnwind
pthread_exit
PyEval_RestoreThread <- interpreter is finalizing
libtorch_python ... c10::TensorImpl::decref_pyobject
c10d::ProcessGroupGloo::runLoop
```
- Non-daemonic workers are started with the `spawn` method, so they run
full interpreter finalization on exit. `fork`/`forkserver` children end
with `os._exit`.
- During finalization a gloo `runLoop` thread can still be releasing the
last reference to a Python-owned tensor. Acquiring the GIL there makes
CPython `pthread_exit` the thread, and the forced unwind through a
`noexcept` frame calls `std::terminate`.
- `destroy_process_group()` alone is not enough: with DeepCompile the
compiled graph keeps the TP group alive (placeholder
`meta['example_value']`), so the gloo threads are not joined.
This reproduces without DeepSpeed: 2 spawned gloo ranks doing one
`all_reduce` and returning without `destroy_process_group()` abort in 3
of 60 runs.
## Fix (test harness workaround)
Spawned workers now exit through a small `mp.Process` subclass that
calls `os._exit` after `_bootstrap`, matching the existing
fork/forkserver behavior. No collective runs at teardown, so a rank
whose test fails cannot leave the others blocked.
## Testing
Local CPU repro (`DS_ACCELERATOR=cpu`, `DS_DISABLE_REUSE_DIST_ENV=1`,
`pytest --forked -m sequential
unit/compile/test_tp_compile.py::TestAutoTPCompileTiedEmbeddings`,
repeated):
- before: 9 aborted runs / 66
- after: 0 / 60
Harness semantics: a test failing on only one rank still reports `Worker
1 exited with code 1` with the assertion; `pytest.skip` inside workers
is still propagated.
Other non-daemonic tests on CPU (`unit/compile/test_tp_compile.py`,
`unit/pipe/test_pipe_module.py`, `unit/comm/test_dist.py`,
`unit/v1/compile/test_compile_zero.py`,
`unit/v1/autotp/test_tp_plan_e2e.py`,
`unit/v1/half_precision/test_keep_in_fp32_modules.py`): 139 passed, 62
skipped.
---------
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>