onnxruntime
8fb1b27f - [CUDA] Add Mamba2Verify CUDA Kernel (#33969)

Commit
3 days ago
[CUDA] Add Mamba2Verify CUDA Kernel (#33969) ## Description Adds `com.microsoft::Mamba2Verify` version 1 (CUDA), a snapshot-aware Mamba2 operator for Nemotron Bolt. Fuses causal convolution, selective recurrence, and gated grouped RMS normalization. A device INT32 selector chooses the starting snapshot; each processed token writes its post-token convolution and SSM snapshots so the caller can select the accepted/final state on the next invocation. The retained contract uses BF16 activations/convolution, FP32 recurrent state, readout and normalization coefficients, head_dim=64, d_state=128, and 1-16 token rows. It preserves state aliasing, trailing snapshots, eight-row specialization, and invalid-selector non-finite output behavior. Longer prefills require chunking. This layer adds no projection, routing, or expert operator and requires no custom-op plugin. **Dependency:** #33963 in [native stack #33967](https://github.com/microsoft/onnxruntime/stack/33967); the eventual landing target is `main`. The slot-aware single-row replay readout contract is included in this introducing layer. ## Summary of Changes ### Schema and Kernel Registration | File | Change | |------|--------| | `onnxruntime/core/graph/contrib_ops/fused_state_space_defs.h` | Define Mamba2Verify inputs/outputs, type constraints, and shape inference. | | `onnxruntime/core/graph/contrib_ops/contrib_defs.cc, onnxruntime/contrib_ops/cuda/cuda_contrib_kernels.cc` | Register the contrib schema and CUDA kernel. | ### Implementation, Tests, and Documentation | File | Change | |------|--------| | `onnxruntime/contrib_ops/cuda/sequence/fused_state_space.cc` | Validate geometry/snapshots and dispatch the CUDA implementation. | | `onnxruntime/contrib_ops/cuda/sequence/fused_state_space_impl.{cu,h}` | Implement convolution, recurrence, readout, normalization, and snapshot writes. | | `onnxruntime/test/contrib_ops/fused_state_space_op_test.cc, onnxruntime/test/contrib_ops/cuda_kernels/fused_state_space_test.cc` | Exercise provider-level state/selector behavior and internal CUDA kernels. | | `docs/contrib_ops/cuda/mamba2_verify.md, docs/ContribOperators.md, docs/OperatorKernels.md` | Describe the operator ABI, snapshot behavior, and CUDA support. | ## Testing ### Optional-output update - `3e40c0d4380572c4edd2aaf47b32883437bba418` - `xbc` (output 3) and `readout` (output 4) are independently optional; omitted intermediates use stream-aware ORT scratch workspace. Required outputs and all existing output positions are unchanged. - The scope remains one Bolt sequence per invocation. The public SSM layout remains `[slots, heads, d_state, head_dim]`. CUDA math, dtypes/cast boundaries, timestep clamping, state selection and the readout/replay guarantee are unchanged. - **PASS:** 19 repository gtests (5 schema, 10 CUDA provider, 4 internal CUDA) plus one fresh-kernel/schema guard: **20 checks, zero failures/skips**, on H100. Profiles record 66 `CUDAExecutionProvider/Mamba2Verify` node executions. - **PASS:** all four intermediate-output combinations, including an empty output-3 placeholder; state aliases, lower/upper invalid selectors and trailing/no-trailing snapshots. Eight capture cases (four combinations at one/eight rows) verify actual graph capture and bitwise replay. - **PASS:** 5 unwrapped CPU-only schema tests, compiled without `USE_CUDA` and run with `CUDA_VISIBLE_DEVICES` empty; zero failures/skips. - **PASS:** configured changed-range clang-format 20.1.8, whitespace checks and scoped official schema/kernel documentation rendering. The kernel-table row remains unchanged; generated schema documentation shows 3-5 outputs. Qualification freshly compiles the operator host/device code, schema and tests against ABI-matched retained ORT 1.32.0 infrastructure. It is **not a clean full-source build, full-CI approval, independent HF parity or full-model acceptance**. The initial mixed-infrastructure-header harness failure was diagnosed in `OpTester::BuildModel` and preserved; no production workaround or threshold relaxation was introduced. CUDA math/internal-test sources are byte-unchanged. ### Historical validation before the optional-output update **Review-fix snapshot:** `4f2bd30fe1f3017213ce9eb620b34ab507f5e392`. Fixed static rank/dimension validation with CPU-only schema tests, dt clamp/large-dt and NaN behavior, strict BF16 ULP/fraction checking and nonuniform 1024-wide normalization. The slot-aware replay correction and `MambaSingleRowReplayPreservesMultiRowReadout` regression are now owned by this introducing PR (moved from #34003), with both host references updated and the single-slot path preserved. The permanent contract is now `docs/contrib_ops/cuda/mamba2_verify.md`. **Owning-layer validation:** 8 provider + 3 CPU schema + 4 actual internal CUDA gtests; zero failures/skips. **Integrated tip regression:** 51 C++ provider/schema tests, 5 actual internal CUDA tests and 2 Python IOBinding tests passed, zero failures/skips. All owning-layer provider cases are retained. The final overlay recompiles the changed CUDA/host/schema/test sources over ABI-matched preserved ORT support; **not a clean full-source build, full-CI approval, independent HF parity or new end-to-end qualification**. Targeted configured clang-format/Ruff and whitespace checks passed. <details> <summary>Historical source provenance moved out of permanent documentation</summary> Origin: #33940, commit `491dc043952066267b2212ba979f0325da318573`, frozen external source `export-nemotron-pr-20261030/src/bolt_ops.cu`, SHA256 `1f587d9eacc356805c37d19e94e7aa4465dd50302576b2ca4b9579d7c5ae8ba5`. This identifies the original retained implementation, not a dependency in the merged tree or a promise that later dispatch fixes are absent. The slot-aware replay correction and current contract are documented in this PR and the permanent operator documentation. This is the Bolt precision contract, not independent HuggingFace model parity. </details> <details> <summary>Historical measurements and qualification evidence (not current-head acceptance)</summary> These previously reported results are retained for provenance. They have not been rerun on the rebased head above; earlier CI/build status and benchmark results do not establish current-head CI, accuracy, or merge readiness. ### Measured CUDA kernel latency - 2026-11-07 BF16 projection/convolution, FP32 SSM state; 128 heads, eight groups, head dimension 64, state size 128, channels 10240, projection stride 18560, four-tap convolution. One fused R-row production launch is compared with R serial single-row production calls with exact state handoff; every timed invocation starts from the same immutable initial state. This compares three fused kernels with 3*R serial kernels. Frozen **Bolt** precision contract: FP32 gate/normalization/gamma followed by BF16 output; this is not an acceptance test for the earlier Lightning cast-before-gamma contract. Conv snapshots, SSM snapshots and xBC are bitwise equal for all three cases in both runs. Y/readout pass normalized RMSE <=0.03 and cosine >=0.999; exact raw metrics are below. The one-row control is approximately unchanged; eight/sixteen-row fusion reduces latency. | Rows | Serial single-row calls us | Fused multi-row call us | Ratio | |---|---|---|---| | 1 | 8.91 | 8.91 | 1.00x | | 8 | 82.98 | 22.17 | 3.74x | | 16 | 167.01 | 41.41 | 4.03x | H100 80GB HBM3, CUDA 12.8.93, driver 580.159.04, GCC 11.4.0. CUDA-event median per dispatch; 64 dispatches per captured CUDA graph, 50 warmups, three graph warmups, and 15 paired timed batches alternating AB/BA. Four rotating allocation banks contain deterministic synthetic weights (seed 617); this is not a cold-cache guarantee or a checkpoint-weight benchmark. Raw p95 values describe batch-average latency, not individual-invocation tails. No profiler runs in the timing window. The assigned GPU had empty admission, one stable compute client during execution, and clean release afterward; clocks were not locked. Two independent 39-case runs passed, including the published replay runner. **Scope:** same-build production CUDA dispatch alternatives, not historical parent-commit binaries. The measured runtime is an isolated CUDA provider relink with preserved core/support, **not a clean full-source ORT build or end-to-end model qualification**. Each measured operator's CUDA source hash matches its owning PR head. A clean source build is a separate replication variant and must be measured again. Kernel-only timings do not establish model TTFT/tok/s, MMLU/LCB, speculative integration, or full-model readiness. [Complete primary/repeat results, numerical metrics, source hashes and benchmark code](https://github.com/microsoft/onnxruntime/pull/33969#issuecomment-6711210138). All 39 cases passed in both supervised runs; full-model work remains paused. No source push, stack change, merge, auto-merge or ready-for-review promotion is included. </details> ## Checklist - [x] Review feedback addressed with focused regressions - [x] Documentation and supported contracts updated - [x] Native stack order and immediate-parent dependencies preserved - [ ] Clean full-source/full-CI qualification (not claimed by focused overlays) --------- Signed-off-by: rui-ren <ruiren1225@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f4d0e1a3-a389-4cc8-9c85-0fd0e3ffce5b
Author
Parents
Loading