onnxruntime
c9a334c2 - Split DirectML CI builds from GPU tests (#34080)

Commit
2 days ago
Split DirectML CI builds from GPU tests (#34080) ### Description <!-- Describe your changes. --> Move DirectML compilation in `.github/workflows/windows_dml.yml` to `ort-github-win-16-amd-scus-ss`, with a dependent test job on the existing `ort-github-win-6-a10-scus-ss` GPU pool. Preserve the existing `Windows GPU DML CI Pipeline` check name on the GPU test job. - CPU: explicit `--update --build --skip_tests`, retaining the original Visual Studio generator, WCOS/BinSkim settings, vcpkg cache, Python 3.12, C# configuration, documentation/delegate validation, NuGet setup, and DirectML wheel packaging. - GPU: install the produced wheel, then run explicit `build.py --test` with all original feature/test flags. No configure, compilation, wheel rebuild, or NuGet restore target runs here. Fix `setup_dml_build` so test-only invocations do not implicitly call `cmake --build --target RESTORE_PACKAGES`. - Hand off the runtime output tree, nested CTest registrations, and any ONNX backend fixtures present in the original build. Relocate producer build/source/Python/CMake paths, retain runtime DLLs/Python packages/test fixtures (including hidden files), and exclude native objects/libraries/debug symbols and dependency build trees. Fail on missing required test binaries, Python/DML runtime files, registrations, or a missing/ambiguous wheel. - Run build-stage regressions on the CPU runner. Coverage includes build-only wheel packaging/C# configuration and test-only execution without build commands. The separate `test_dml_test_artifacts.py` module was removed as requested; the production `dml_test_artifacts.py` helper remains. C# scope: the existing workflow uses `--build_csharp` but not `--build_nuget`. In the current build driver/CMake code this generates C# build properties; it does not invoke NuGet packaging or `dotnet test`. That behavior and the delegate-attribute validation are preserved, not replaced with a different C# test surface. **Observed performance:** the [successful split run](https://github.com/microsoft/onnxruntime/actions/runs/42848703495) reduced GPU-runner occupancy by approximately **45%** and end-to-end execution time by approximately **24%**, compared with the average of three successful main runs ([42838643765](https://github.com/microsoft/onnxruntime/actions/runs/42838643765), [42861308845](https://github.com/microsoft/onnxruntime/actions/runs/42861308845), [42863264161](https://github.com/microsoft/onnxruntime/actions/runs/42863264161)). | Metric | Before: combined GPU job (three-run average) | After: split jobs | Difference | | --- | --- | --- | --- | | GPU-runner time | 1h 48m 53s | 1h 00m 21s | 48m 32s less (45%) | | End-to-end execution | 1h 48m 53s | 1h 22m 29s | 26m 24s faster (24%) | | CPU-runner time | None | 21m 22s | Build moved off GPU | The CPU build step took **16m 18s**; the GPU test step took **56m 31s**. Artifact staging, upload, download, and relocation took **1m 19s** combined. End-to-end is measured from the first job start through the final job completion, including the split's 46-second inter-job gap but excluding initial queue time. This is an observed comparison, not a controlled benchmark; there is one successful split sample so far. **Workflow scan:** examined all 47 workflow YAML files and all 13 local composite actions. The only active combined DirectML build/test workflow is `windows_dml.yml`. The DirectML-capable runtime build in `windows_gpu_doc_gen.yml` already runs on the CPU pool with a dependent GPU documentation-generation job. Other DirectML strings in CPU workflows are package cleanup/reference strings, not DML-enabled build variants. Other candidates found, deliberately left unchanged: | Path | Job / remaining GPU-side work | | --- | --- | | `.github/workflows/windows_webgpu.yml` | `webgpu_static_plugin_build_x64_RelWithDebInfo`: combined native build/test on GPU (separate work already planned; this PR does not depend on that branch). | | `.github/workflows/windows_webgpu.yml` | `webgpu_external_dawn_build_x64_RelWithDebInfo`: configure/build, external-Dawn host tests, and package export on GPU. | | `.github/workflows/windows_webgpu.yml` | `webgpu_dawn_package_build_x64_RelWithDebInfo`: package-backed ORT/plugin builds, negative configure checks, and GPU host tests. | | `.github/workflows/windows-web-ci-workflow.yml` | `build_onnxruntime_web`: WASM is already downloaded from a separate build, but NPM packaging and browser tests still share the GPU runner. This is a JS packaging candidate, not another unsplit native WASM build. | Windows CUDA/no-cuDNN/CUDA-plugin/TensorRT and the corresponding Linux workflows already use CPU-build/GPU-test handoffs. Windows WebGPU built-in/shared-plugin variants also already have split jobs. No unrelated workflow/composite action, compiler-cache adoption, or generator migration is included. **Validation** (isolated local Python environment; commands run from the repository root): - `.\.venv\Scripts\python.exe -m unittest tools.ci_build.test_build_args -q` — passed, 24 tests after removing the artifact test module. - `.\.venv\Scripts\python.exe -m ruff check tools\ci_build\build.py tools\ci_build\test_build_args.py tools\ci_build\github\windows\dml_test_artifacts.py` — passed. - `.\.venv\Scripts\python.exe -m ruff format --check tools\ci_build\build.py tools\ci_build\test_build_args.py tools\ci_build\github\windows\dml_test_artifacts.py` — passed, three files already formatted. - `git diff --check` and `git diff --cached --check` — passed before commit. - Parsed the revised DirectML workflow with PyYAML and asserted the production staging/prepare helper calls remain, the deleted test module is absent, and the remaining build-stage regression invocation is intact — passed. - Initial split validation also parsed all 60 workflow/composite YAML files and all six DirectML PowerShell steps. Assertions comparing original build options (only stage/skip flags differ), triggers/concurrency/environment/check name, runner/dependency wiring, artifact identity/guards, wheel ordering, and unchanged unrelated workflows/actions passed. - Locally reproduced the telemetry launcher failure with a missing producer CMake path, then confirmed both nested CTest launchers executed after relocation, including forward-slash and escaped Windows paths. Missing producer/consumer CMake checks failed explicitly as intended. - [DirectML CI run 42848703495](https://github.com/microsoft/onnxruntime/actions/runs/42848703495) at commit `0b1507622bc29e9cd1e757e17a65a8653b08bc96` — passed both the CPU build and GPU test jobs, verifying the artifact handoff and fixing the four telemetry shutdown tests that previously retained the CPU runner's absolute CMake path. Full native DirectML compilation, production wheel installation, GPU test execution, and the actual GitHub artifact transfer were not run locally; they were verified by the successful CI run above. No workflow was manually dispatched and no live cache was modified. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Use the established 16-core Windows CPU pool for compilation instead of occupying the six-core GPU pool. Reserve the GPU runner for backend execution while retaining the existing DirectML CI coverage and validation settings. The observed successful run saves approximately 49 GPU-runner minutes per invocation and completes approximately 26 minutes sooner than the recent main-run average. This is an independent PR based on main, separate from the WebGPU/static-plugin and macOS caching work. --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Author
Parents
Loading