Major Refactoring of Azure DevOps Pipelines (#26008)
This pull request introduces a significant refactoring of the Azure
Pipelines CI/CD infrastructure. The primary goals of these changes are
to:
1. Solve the problem that vcpkg/cmake version can get changed when the
CI build machine image changes, which can make pipeline suddenly broken
and interrupt our release process.
2. Reduce the `Zip-Nuget-Java-Nodejs Packaging Pipeline`'s running time
by changing how macOS universal2 binaries are built.
3. Add the support for RC releases for Java packages.
#### Key Changes:
**1. Standardized Build Tool Setup (`setup-build-tools.yml`)**
A new reusable template, `setup-build-tools.yml`, has been created to
centralize the setup of essential build tools.
* **Pinned Dependencies:** This new template allows us to pin specific
versions of `cmake` and `vcpkg`, which was not previously possible. This
ensures a more stable and predictable build environment across all
pipelines.
* **Reduced Redundancy:** By consolidating the setup logic into a single
template, we have significantly reduced code duplication across numerous
pipeline files, making them cleaner and easier to maintain.
Currently this file is only used in macOS and Windows pipelines since
most Linux pipelines use docker to manage their environment.
**2. Reworked macOS Universal Binary Build Process**
The methodology for building macOS `universal2` binaries has been
fundamentally changed to improve reliability and flexibility.
* **Python Packaging:** The Python packaging pipeline will no longer
produce `universal2` wheels. Instead, it will generate separate wheels
for `x86_64` and `arm64` architectures.
* **NuGet C-API Packaging:** The NuGet C-API pipeline has been updated
to first build the `x86_64` and `arm64` binaries independently. These
single-architecture binaries are then combined to create the final
universal package, rather than building a `universal2` package in a
single pass.
The change is made mainly because:
- Building for both ARCHs in a single pass is too slow, which may take
about 5 hours in the ADO machine pool.
- A lot of MLAS features are disabled when ORT is built in such a way.
**3. Java Packaging and Testing Overhaul**
The Download_Java_Tools stage in "Zip-Nuget-Java-Nodejs Packaging
Pipeline" is deleted because it is no longer used. Previously it was
added to reduce the times of downloading the same java packages again
and again , which was to reduce download errors. Now we have setup a
private ADO feed for this.
Besides, there are some major changes to the pipeline that:
1. MD5 and SHA1 checksum files are provided along with the java package
files instead of SHA256. This is because Sonatype's official docs says
MD5/SHA1 checkcums are required while the others are optional. See:
https://central.sonatype.org/publish/requirements/#supply-javadoc-and-sources
. Now the publishing would fail if we don't have the MD5/SHA1 checksum
files.
2. The format of the checksum files is changed. Previously we used
Linux's sha256sum command to generate such files, so each checksum file
contains a hash value and a filename in the file content. However, it
was not the expected format. Sonatype expects that the file only
contains a hash value. This PR fixes the issue.
3. A few powershell scripts were rewritten in python to improve error
check and robustness
4. Added the support for generating RC packages. Previously we had to
manually modify the version numbers and manually do GPG sign.
5. Two new files `jar-packaging.yml` and `setup-maven.yml` were added.
We will use maven to fetch dependent packages(instead of directly HTTP
fetching) to improve supply chain security, because maven allows us
using a private feed to do so.
**4. Dockerfile Enhancements**
The Dockerfiles used in the CI have been updated to use a `BASEIMAGE`
argument. This makes them more flexible, allowing the base image to be
specified at build time, which simplifies maintenance and updates. It
will allow us to using different base image repos in different CI
environments. In the future we will change the Github Actions to only
fetch base images from public docker repos. Meanwhile, ADO packaging
pipelines will continue to use private repos.
**5. Improved Release Management**
The run_packaging_pipelines.py script has been updated to provide more
robust and explicit control over
the package versioning for different build scenarios. This clarifies the
process for generating nightly,
release candidate (RC), and final release packages.
The script now handles three distinct cases for package versioning:
* Nightly Packages: For regular CI builds (e.g., on the main branch),
the script triggers the packaging
pipelines in "nightly" mode. This sets the IsReleaseBuild parameter to
false and results in packages with
a development version suffix (e.g., 1.2.3-dev-20250909-abcdef).
* Release Candidate (RC) Builds: To create a pre-release or RC build,
the script is run with the
--build-mode release flag, along with the --pre-release-suffix-string
(e.g., rc) and
--pre-release-suffix-number (e.g., 1) arguments. This sets the
IsReleaseBuild parameter to true and
passes the suffix information to the pipelines, resulting in a
semantically versioned pre-release package
(e.g., 1.2.3-rc.1).
* Final Release Builds: For a final release, the script is run with
--build-mode release without any
pre-release suffix arguments. This sets IsReleaseBuild to true, and the
resulting package will have a
clean, final version number (e.g., 1.2.3) based on the VERSION_NUMBER
file.
Please note:
- Java packages still only support the second and third mode.
- Python packages only support the first and the last mode.
