onnxruntime

dd32f359 - Fix libcudart.so.13 hard dependency in pybind module breaking import on CPU-only Linux (#29590)

Fix libcudart.so.13 hard dependency in pybind module breaking import on CPU-only Linux (#29590) ### Description `onnxruntime-gpu` 1.27 introduced a hard `NEEDED libcudart.so.13` entry in `onnxruntime_pybind11_state.so`, causing `ImportError` at `import onnxruntime` on CPU-only Linux machines — before any provider is selected. **Root cause:** `cmake/onnxruntime_python.cmake` was changed to compile `fpA_intB_gemm_adaptor.cu` and `fpA_intB_gemm_preprocessors_impl.cu` directly into `onnxruntime_pybind11_state.so` and link `CUDA::cudart` (dynamic). This embeds a load-time CUDA dependency in the Python module itself. **Fix:** Move the CUDA weight-preprocessing entry point (`pack_weights_for_cuda_mixed_gemm`) out of the main pybind module and into a **standalone extension module**, `onnxruntime_cuda_quant_preprocess`, that links `CUDA::cudart` on its own. The main `onnxruntime_pybind11_state.so` no longer compiles or links any CUDA code, so `import onnxruntime` has no `libcudart` dependency. The new module is imported **lazily** by `onnxruntime/python/tools/quantization/cuda_quantizer.py` only when weight prepacking is actually requested — never at `import onnxruntime` time. These preprocessing APIs are **offline-only** helpers: they are used by quantization tooling and model builders to produce prepacked weight initializers ahead of time, and are not part of the inference runtime hot path. Because nothing in the runtime imports them, isolating them into a separate, on-demand DLL has no runtime cost and cleanly keeps CUDA out of the base `import onnxruntime` path. **Why not the provider bridge:** An earlier iteration routed the call through the `ProviderInfo_CUDA` virtual interface (`TryGetProviderInfo_CUDA()`). That does not work for the CUDA-EP-as-plugin build (`onnxruntime_BUILD_CUDA_EP_AS_PLUGIN=ON`): `cuda_provider_factory.cc` is excluded from the plugin sources and there is no provider bridge, so `TryGetProviderInfo_CUDA()` returns `nullptr` and the call throws. The standalone module has no such dependency and works for **both** the legacy in-tree CUDA EP build and the plugin build. ### Key Changes | File | Change | |---|---| | `onnxruntime/python/onnxruntime_pybind_cuda_quant.cc` | **New.** Self-contained `pack_weights_for_cuda_mixed_gemm` (device malloc + transpose/convert + arch permutation) and a `PYBIND11_MODULE(onnxruntime_cuda_quant_preprocess, …)` entry point. | | `cmake/onnxruntime_python.cmake` | Add the `onnxruntime_cuda_quant_preprocess` module target (built when `onnxruntime_USE_CUDA AND NOT WIN32`, compiling the two `fpA_intB` `.cu` files + `CUDA::cudart` + cutlass, hidden visibility) and copy it into `onnxruntime/capi/`. Main pybind module keeps no CUDA sources/links. | | `onnxruntime/python/onnxruntime_pybind_quant.cc` | Remove the `USE_CUDA` `PackWeightsForMixedGemm` and its registration. The CPU-only `pack_fp4_weights_for_cuda_moe_gemm` stays in the main module. | | `onnxruntime/core/providers/cuda/cuda_provider_factory.{h,cc}` | Revert the `PackWeightsForMixedGemm` `ProviderInfo_CUDA` addition (no longer needed; absent in plugin builds). | | `onnxruntime/python/tools/quantization/cuda_quantizer.py` | `_get_pack_weights_for_cuda_mixed_gemm()` now imports `onnxruntime.capi.onnxruntime_cuda_quant_preprocess` lazily; add `has_cuda_weight_prepacking()` capability helper. | | `setup.py` | Package `onnxruntime_cuda_quant_preprocess.so` in the Linux/macOS wheels. | | `onnxruntime/test/python/quantization/test_op_matmulnbits_prepacked_cuda.py` | Point the prepacked-weight parity test and its skip guard at the new module. | | `docs/contrib_ops/cuda/matmul_nbits.md` | Update the offline-packer code snippets to import the new module. | ### Motivation and Context `import onnxruntime` must succeed on CPU-only machines even when the GPU wheel is installed. CUDA dependency errors should surface only when a CUDA provider is explicitly loaded/selected, or when offline CUDA weight prepacking is explicitly requested. This restores the 1.26 behavior where `onnxruntime_pybind11_state.so` had no `NEEDED libcudart.so.*` entry, and — unlike the provider-bridge approach — it also works in the CUDA-EP-as-plugin build. ### Testing Notes - Built both modules in the CUDA build; `readelf -d onnxruntime_pybind11_state.so` shows **no** `libcudart` `NEEDED` entry, while `onnxruntime_cuda_quant_preprocess.so` has `NEEDED libcudart.so.13`. - `import onnxruntime` and lazy loading of `onnxruntime.capi.onnxruntime_cuda_quant_preprocess` both succeed; `has_cuda_weight_prepacking()` returns `True` on a CUDA machine. - `test_op_matmulnbits_prepacked_cuda.py` passes (INT4/INT8 prepacked-vs-runtime parity), confirming the relocated packer produces byte-identical prepacked weights. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com>