Split DirectML CI builds from GPU tests (#34080)
### Description
<!-- Describe your changes. -->
Move DirectML compilation in `.github/workflows/windows_dml.yml` to
`ort-github-win-16-amd-scus-ss`, with a dependent test job on the
existing `ort-github-win-6-a10-scus-ss` GPU pool. Preserve the existing
`Windows GPU DML CI Pipeline` check name on the GPU test job.
- CPU: explicit `--update --build --skip_tests`, retaining the original
Visual Studio generator, WCOS/BinSkim settings, vcpkg cache, Python
3.12, C# configuration, documentation/delegate validation, NuGet setup,
and DirectML wheel packaging.
- GPU: install the produced wheel, then run explicit `build.py --test`
with all original feature/test flags. No configure, compilation, wheel
rebuild, or NuGet restore target runs here. Fix `setup_dml_build` so
test-only invocations do not implicitly call `cmake --build --target
RESTORE_PACKAGES`.
- Hand off the runtime output tree, nested CTest registrations, and any
ONNX backend fixtures present in the original build. Relocate producer
build/source/Python/CMake paths, retain runtime DLLs/Python
packages/test fixtures (including hidden files), and exclude native
objects/libraries/debug symbols and dependency build trees. Fail on
missing required test binaries, Python/DML runtime files, registrations,
or a missing/ambiguous wheel.
- Run build-stage regressions on the CPU runner. Coverage includes
build-only wheel packaging/C# configuration and test-only execution
without build commands. The separate `test_dml_test_artifacts.py` module
was removed as requested; the production `dml_test_artifacts.py` helper
remains.
C# scope: the existing workflow uses `--build_csharp` but not
`--build_nuget`. In the current build driver/CMake code this generates
C# build properties; it does not invoke NuGet packaging or `dotnet
test`. That behavior and the delegate-attribute validation are
preserved, not replaced with a different C# test surface.
**Observed performance:** the [successful split
run](https://github.com/microsoft/onnxruntime/actions/runs/42848703495)
reduced GPU-runner occupancy by approximately **45%** and end-to-end
execution time by approximately **24%**, compared with the average of
three successful main runs
([42838643765](https://github.com/microsoft/onnxruntime/actions/runs/42838643765),
[42861308845](https://github.com/microsoft/onnxruntime/actions/runs/42861308845),
[42863264161](https://github.com/microsoft/onnxruntime/actions/runs/42863264161)).
| Metric | Before: combined GPU job (three-run average) | After: split
jobs | Difference |
| --- | --- | --- | --- |
| GPU-runner time | 1h 48m 53s | 1h 00m 21s | 48m 32s less (45%) |
| End-to-end execution | 1h 48m 53s | 1h 22m 29s | 26m 24s faster (24%)
|
| CPU-runner time | None | 21m 22s | Build moved off GPU |
The CPU build step took **16m 18s**; the GPU test step took **56m 31s**.
Artifact staging, upload, download, and relocation took **1m 19s**
combined. End-to-end is measured from the first job start through the
final job completion, including the split's 46-second inter-job gap but
excluding initial queue time. This is an observed comparison, not a
controlled benchmark; there is one successful split sample so far.
**Workflow scan:** examined all 47 workflow YAML files and all 13 local
composite actions. The only active combined DirectML build/test workflow
is `windows_dml.yml`. The DirectML-capable runtime build in
`windows_gpu_doc_gen.yml` already runs on the CPU pool with a dependent
GPU documentation-generation job. Other DirectML strings in CPU
workflows are package cleanup/reference strings, not DML-enabled build
variants.
Other candidates found, deliberately left unchanged:
| Path | Job / remaining GPU-side work |
| --- | --- |
| `.github/workflows/windows_webgpu.yml` |
`webgpu_static_plugin_build_x64_RelWithDebInfo`: combined native
build/test on GPU (separate work already planned; this PR does not
depend on that branch). |
| `.github/workflows/windows_webgpu.yml` |
`webgpu_external_dawn_build_x64_RelWithDebInfo`: configure/build,
external-Dawn host tests, and package export on GPU. |
| `.github/workflows/windows_webgpu.yml` |
`webgpu_dawn_package_build_x64_RelWithDebInfo`: package-backed
ORT/plugin builds, negative configure checks, and GPU host tests. |
| `.github/workflows/windows-web-ci-workflow.yml` |
`build_onnxruntime_web`: WASM is already downloaded from a separate
build, but NPM packaging and browser tests still share the GPU runner.
This is a JS packaging candidate, not another unsplit native WASM build.
|
Windows CUDA/no-cuDNN/CUDA-plugin/TensorRT and the corresponding Linux
workflows already use CPU-build/GPU-test handoffs. Windows WebGPU
built-in/shared-plugin variants also already have split jobs. No
unrelated workflow/composite action, compiler-cache adoption, or
generator migration is included.
**Validation** (isolated local Python environment; commands run from the
repository root):
- `.\.venv\Scripts\python.exe -m unittest tools.ci_build.test_build_args
-q` — passed, 24 tests after removing the artifact test module.
- `.\.venv\Scripts\python.exe -m ruff check tools\ci_build\build.py
tools\ci_build\test_build_args.py
tools\ci_build\github\windows\dml_test_artifacts.py` — passed.
- `.\.venv\Scripts\python.exe -m ruff format --check
tools\ci_build\build.py tools\ci_build\test_build_args.py
tools\ci_build\github\windows\dml_test_artifacts.py` — passed, three
files already formatted.
- `git diff --check` and `git diff --cached --check` — passed before
commit.
- Parsed the revised DirectML workflow with PyYAML and asserted the
production staging/prepare helper calls remain, the deleted test module
is absent, and the remaining build-stage regression invocation is intact
— passed.
- Initial split validation also parsed all 60 workflow/composite YAML
files and all six DirectML PowerShell steps. Assertions comparing
original build options (only stage/skip flags differ),
triggers/concurrency/environment/check name, runner/dependency wiring,
artifact identity/guards, wheel ordering, and unchanged unrelated
workflows/actions passed.
- Locally reproduced the telemetry launcher failure with a missing
producer CMake path, then confirmed both nested CTest launchers executed
after relocation, including forward-slash and escaped Windows paths.
Missing producer/consumer CMake checks failed explicitly as intended.
- [DirectML CI run
42848703495](https://github.com/microsoft/onnxruntime/actions/runs/42848703495)
at commit `0b1507622bc29e9cd1e757e17a65a8653b08bc96` — passed both the
CPU build and GPU test jobs, verifying the artifact handoff and fixing
the four telemetry shutdown tests that previously retained the CPU
runner's absolute CMake path.
Full native DirectML compilation, production wheel installation, GPU
test execution, and the actual GitHub artifact transfer were not run
locally; they were verified by the successful CI run above. No workflow
was manually dispatched and no live cache was modified.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Use the established 16-core Windows CPU pool for compilation instead of
occupying the six-core GPU pool. Reserve the GPU runner for backend
execution while retaining the existing DirectML CI coverage and
validation settings. The observed successful run saves approximately 49
GPU-runner minutes per invocation and completes approximately 26 minutes
sooner than the recent main-run average. This is an independent PR based
on main, separate from the WebGPU/static-plugin and macOS caching work.
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>