onnxruntime

291311e7 - Fix CUDA 13 nuget packaging pipeline failures and enable CUDA 13.3 builds (#28736)

Fix CUDA 13 nuget packaging pipeline failures and enable CUDA 13.3 builds (#28736) ## Description Fixes OOM failures in Windows CUDA 13 nuget packaging pipelines caused by excessive `nvcc_threads` parallelism, and resolves multiple compilation errors when building with CUDA 13.2 and 13.3 (new NVCC code generation, stricter Abseil templates, MSVC alignment incompatibilities with TMA host stubs). ## Summary of Changes ### CI/Pipeline: OOM Fix (nvcc_threads reduction) | File | Change | |------|--------| | `tools/ci_build/github/azure-pipelines/custom-nuget-packaging-pipeline.yml` | `--nvcc_threads 4 --flash_nvcc_threads 2` → `1`/`1` | | `tools/ci_build/github/azure-pipelines/stages/nuget-win-cuda-packaging-stage.yml` | Same reduction for CUDA and TensorRT packaging | | `tools/ci_build/github/azure-pipelines/stages/plugin-win-cuda-stage.yml` | Same reduction for plugin build and test | | `tools/ci_build/github/azure-pipelines/stages/py-win-gpu-stage.yml` | Same reduction for Python GPU stage | | `tools/ci_build/github/linux/build_cuda_c_api_package.sh` | Same reduction for Linux C API packaging | | `tools/ci_build/github/linux/build_cuda_plugin_package.sh` | `--nvcc_threads 2` → `1` | | `tools/ci_build/github/linux/build_linux_python_package.sh` | Same reduction for Linux Python packaging | ### CUDA 13.2/13.3 Compilation Fixes | File | Change | |------|--------| | `cmake/external/cuda_configuration.cmake` | Fix architecture detection for suffixed values (e.g. `90a`, `100f`) — use regex match instead of exact `IN_LIST` | | `cmake/patches/abseil/absl_cuda_warnings.patch` | Patch Abseil `friend class` declarations that CUDA 13 rejects (`MixingHashState::HashStateBase` → `HashStateBase<MixingHashState>`) | | `cmake/vcpkg-ports/abseil/absl_cuda_warnings.patch` | Mirror of above for vcpkg-based builds | | `cmake/onnxruntime_providers_cuda.cmake` | Add `/Zc:preprocessor` for CXX and CUDA on MSVC | | `cmake/onnxruntime_providers_cuda_plugin.cmake` | Same `/Zc:preprocessor` flags + add `${onnxruntime_EXTERNAL_DEPENDENCIES}` to ensure generated headers exist | | `cmake/onnxruntime_cuda_source_filters.cmake` | Add external dependencies to plugin object libraries | | `onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/fmha_launch_template.h` | Add `static_cast<uint8_t>` to fix narrowing conversion warning | | `onnxruntime/contrib_ops/cuda/quantization/matmul_4bits.cu` | Replace bitmask-based loop bound with modulo (avoid UB with non-power-of-2 unroll) | | `onnxruntime/core/providers/cuda/tensor/concat_impl.cu` | Add `const` to explicit template instantiation to match declaration | | `onnxruntime/core/providers/cuda/tensor/resize_antialias_impl.cu` | Add `ORT_UNUSED_PARAMETER(batch_size)` to suppress unused-parameter warning | ### MSVC TMA Grouped MoE Kernel Workaround | File | Change | |------|--------| | `cmake/onnxruntime_providers_cuda.cmake` | Wrap `COMPILE_HOPPER_TMA_GROUPED_GEMMS` and `COMPILE_BLACKWELL_SM120_TMA_GROUPED_GEMMS` in `if(NOT MSVC)` | | `cmake/onnxruntime_providers_cuda_plugin.cmake` | Same MSVC guards for the plugin build path | | `onnxruntime/contrib_ops/cuda/llm/moe_gemm/moe_gemm_template_dispatch.h` | Add compile-feature constants, `isTmaWarpSpecializedGroupedGemmCompiledForSm()`, early-out in `getTmaWarpSpecializedConfigs()` (throws for QMoE, returns empty for standard MoE), and guard in `supportsTmaWarpSpecialized()` | | `onnxruntime/contrib_ops/cuda/llm/moe_gemm/moe_gemm_kernels.h` | Remove unused `getBlackwellConfigs`/`getHopperConfigs` declarations | | `docs/contrib_ops/cuda/moe_qmoe.md` | Add §14.1 documenting the MSVC limitation and fallback behavior | ### Infrastructure / CI Fixes | File | Change | |------|--------| | `tools/ci_build/github/linux/build_nodejs_package.sh` | Add `--network=host` to docker run | | `.github/workflows/windows_cuda.yml` | Remove `1ES.ImageOverride` from test pool | | `.github/workflows/windows_cuda_plugin.yml` | Same ImageOverride removal | ## Motivation and Context The CUDA 13 nuget packaging pipelines on Windows were failing with two classes of issues: 1. **OOM during compilation**: With `--nvcc_threads 4` and multi-architecture builds (80, 86, 89, 90, 100, 120), the packaging VMs run out of memory. Reducing to `--nvcc_threads 1 --flash_nvcc_threads 1` prevents this. 2. **CUDA 13.2/13.3 compiler incompatibilities**: - NVCC 13.x generates host stubs that pass TMA descriptor parameters (with `alignas(128)`) by value. MSVC rejects this with error `C2719`. The workaround is to disable grouped TMA MoE kernel compilation on MSVC, with proper runtime detection so standard MoE gracefully falls back to SM80 Ampere kernels while QMoE modes that truly require TMA produce a clear error. - Abseil's `friend class` forward declarations use a non-standard form that CUDA 13 rejects. - Architecture detection in CMake failed for suffixed arch specs (e.g., `90a`). - Various minor issues: narrowing conversions, unused parameters, const mismatches. ## Testing - Windows CUDA 13 nuget packaging pipeline should complete without OOM or C2719 errors - Linux CUDA packaging pipelines should complete with reduced nvcc_threads - Standard MoE on SM90+ under MSVC build falls back to SM80 Ampere grouped GEMM - QMoE FP4/block-scaled modes under MSVC build produce a clear `ORT_THROW` error rather than silent misbehavior - Existing CUDA 12.x builds are unaffected (MSVC guard is specific to the `if(NOT MSVC)` pattern) ## Checklist - [ ] Tests added/updated - [x] Documentation updated (docs/contrib_ops/cuda/moe_qmoe.md §14.1) - [x] No breaking changes (MSVC builds never had working TMA grouped MoE kernels with CUDA 13) - [ ] CI passes