Add a v1 multi-rank CPU suite (manual dispatch first) (#8926)
## What
Adds `cpu-torch-latest-v1-multirank`, a CPU suite for the
**`tests/unit/v1`** multi-rank tests — the one gap left between the
existing CPU workflows:
| Workflow | Covers |
|---|---|
| `cpu-torch-latest` | all of `unit/` at `world_size == 1` (the device
gate skips higher ranks) |
| `cpu-torch-multi-latest` (nightly) | multi-rank for the non-v1 half +
the `sequential` marker |
| **`cpu-torch-latest-v1-multirank` (this PR)** | **v1 multi-rank**
(`world_size` 2-4) |
Mechanically it is the same recipe the nightly suite uses:
`LOCAL_SIZE=4` lets the per-device gate admit the multi-rank tests, and
`DS_MULTIRANK_ONLY=1` deselects everything else, so the job spends its
time on v1 multi-rank tests only.
## Selection
Test selection reuses the existing Modal diff-based picker
(`ci/tests_fetcher.py`): the new `WORKFLOWS` entry scopes the suite to
`tests/unit/v1`, so a PR only runs the v1 files it could affect. The
`collect-tests` job writes `ci/.test_selection/test_list.txt` and the
test job consumes it; `mode=none` skips the run entirely. No test lists
are maintained by hand.
## Rollout: manual dispatch first
The trigger is `workflow_dispatch` only, deliberately. This suite has
never run in CI, and putting an unvalidated ~1 h suite in front of every
pull request (or into branch protection) before it has a green run would
block PRs. Once it is green across a few real PRs, `pull_request:` goes
back into the `on:` block and the check can be registered.
## Harness changes this suite needs (included here)
The v1 multi-rank tests rely on two fixes that used to live on the
work-in-progress #8381 branch; they are part of this PR so the first run
can be green:
- `tests/unit/v1/zero/test_zero_autocast.py`: the DDP `device_ids`
capability probe — without it, all 24 DDP autocast variants fail on CPU
with `ValueError: DistributedDataParallel device_ids and output_device
arguments ...`. The fp16 parity variants are **skipped** when the
accelerator reports no fp16 support (`is_fp16_supported()` false) rather
than widening the tolerance, so no assertion is relaxed.
- `tests/unit/common.py`: shm-segment key isolation/cleanup, JIT warm-up
with stale-baton cleanup, per-rank faulthandler, and master-port
uniquification — without them, concurrent pools share shm segments and
can corrupt data or wedge.
Verified on CPU, 2 ranks: bf16 ZeRO-3 passes, fp16 variants skip when
fp16 is unavailable, and no `/dev/shm` segments are left behind.
## Validation
- `python3 ci/test_tests_fetcher.py`: 27/27 (adds a registry test for
the new `WORKFLOWS` entry)
- `ci/tests_fetcher.py --workflow cpu-torch-latest-v1-multirank`:
`mode=all` (-> `tests/unit/v1`) with no base, `mode=none` with no
impacted tests; `ci/torch_latest.py validate-selection` passes
- Both files pass pre-commit (yapf, flake8, yaml, license)
## AI assistance
Prepared with AI assistance (DeepSeek Harness agent) under human review;
the submitter has read every changed line.
Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
---------
Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
