onnxruntime
42518d89 - [CUDA] Add persistent fpA_intB MatMulNBits tactic autotune cache (#29588)

Commit
34 days ago
[CUDA] Add persistent fpA_intB MatMulNBits tactic autotune cache (#29588) ### Description Adds an **opt-in, hardware-keyed on-disk tactic cache** for the fpA_intB weight-only `MatMulNBits` path, plus a reduced first-time profiling sweep with lazy fallback. Persistence is off by default: with no cache location configured, behavior is unchanged (in-process only). **Cache subsystem** (`gemm_tactic_cache.{h,cc}`) - `HardwareSignature` (GPU name, SM, CUDA runtime, ORT version; commit/build config/driver/MP count recorded for diagnostics) names and guards cache files, so tactics are never reused across incompatible GPUs/toolkits/ORT versions. A strict `tactic_selection_version` header field rejects caches tuned under an older selection policy, even when the ORT version is unchanged. - TSV format: percent-encoded free-text fields, a column-name header (appended columns are backward/forward compatible), file locking + atomic write through a per-writer temp file and rename (`MoveFileExW` on Windows; paths are UTF-8), in-memory merge on flush so concurrent sessions tuning different shapes don't lose updates, and creation of a missing cache directory on first write. - `MatMulNBitsTacticCache` keyed by `(N16b, K, act dtype, weight type, bits, block_size, zero-points, gemv_enabled, packing_sm)`. `packing_sm` keeps the SM80-compat and native-SM90 tactics for the same shape distinct on one device (e.g. H200). 2-, 4-, and 8-bit weights are keyed separately. **Profiler** (`gemm_profiler.h`, `fpA_intB_gemm_profiler.{h,cc}`) - Reduced initial sweep to a small M bucket set (`{1,2,4,…,2048}` default; override with `ep.cuda.fpa_intb_profile_m` / `ORT_FPA_INTB_PROFILE_M`). `getBestConfigOrProfile` lazily profiles any missing bucket on first use. - Persistent-cache load/store hooks are no-ops unless a cache is attached. - CUTLASS tactics loaded from disk are validated against the runner's current `getConfigs()` and dropped (re-profiled) if incompatible. **Wiring** (`matmul_nbits.{h,cc}`, `cuda_execution_provider.cc`, plugin `cuda_ep.cc`) - Cache location: session config (`ep.cuda.gemm_tactic_cache_prefix` / `_dir`) takes precedence over env (`ORT_CUDA_GEMM_TACTIC_CACHE_PREFIX` / `_DIR`); a prefix wins over a dir. - Process-global caches are keyed by resolved file path; the file is loaded once per location (not once per node), and the signature is built from the kernel's cached device properties. - The process-global in-memory profile map (`GemmIdCore`) is keyed by the same quantization fields as the persistent key (bits, block size, zero points, GEMV support), so kernels with different quantization never share tactics. - Construction-time sweeps and lazily profiled buckets are staged in memory as soon as they are profiled, and all disk writes are batched at CUDA EP teardown (`~CUDAExecutionProvider`, or `~CudaEp` for the plugin EP), so there is no disk I/O on the inference path. - Lazy profiling is skipped while the compute stream is capturing a CUDA graph; run a warmup inference before capture. **Offline tuning tool**: `onnxruntime/python/tools/fpa_intb_tune.py` (uses session config entries; releases the session before summarizing so lazily tuned buckets are flushed). ### Testing - `gemm_tactic_cache_test.cc` (built into `onnxruntime_providers_cuda_ut` with `onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON`, run via `--gtest_filter=CUDA_EP_Unittest.All`): TSV round-trip and malformed escapes, SM80/SM90 config serialize/parse, null tactic, store/load, signature-mismatch rejection, non-strict commit/build config, selection-version rejection, appended-column tolerance, concurrent flush-merge, missing-directory creation, file path resolution, and `GemmIdCore` separation of quantization variants. - `*MatMulNBits*` CUDA integration tests. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Author
Parents
Loading