[CUDA] Add Mamba2Verify CUDA Kernel (#33969)
## Description
Adds `com.microsoft::Mamba2Verify` version 1 (CUDA), a snapshot-aware
Mamba2 operator
for Nemotron Bolt. Fuses causal convolution, selective recurrence, and
gated grouped
RMS normalization. A device INT32 selector chooses the starting
snapshot; each
processed token writes its post-token convolution and SSM snapshots so
the caller
can select the accepted/final state on the next invocation.
The retained contract uses BF16 activations/convolution, FP32 recurrent
state,
readout and normalization coefficients, head_dim=64, d_state=128, and
1-16 token
rows. It preserves state aliasing, trailing snapshots, eight-row
specialization,
and invalid-selector non-finite output behavior. Longer prefills require
chunking.
This layer adds no projection, routing, or expert operator and requires
no custom-op plugin.
**Dependency:** #33963 in
[native stack
#33967](https://github.com/microsoft/onnxruntime/stack/33967);
the eventual landing target is `main`. The slot-aware single-row replay
readout contract is included in this introducing layer.
## Summary of Changes
### Schema and Kernel Registration
| File | Change |
|------|--------|
| `onnxruntime/core/graph/contrib_ops/fused_state_space_defs.h` | Define
Mamba2Verify inputs/outputs, type constraints, and shape inference. |
| `onnxruntime/core/graph/contrib_ops/contrib_defs.cc,
onnxruntime/contrib_ops/cuda/cuda_contrib_kernels.cc` | Register the
contrib schema and CUDA kernel. |
### Implementation, Tests, and Documentation
| File | Change |
|------|--------|
| `onnxruntime/contrib_ops/cuda/sequence/fused_state_space.cc` |
Validate geometry/snapshots and dispatch the CUDA implementation. |
| `onnxruntime/contrib_ops/cuda/sequence/fused_state_space_impl.{cu,h}`
| Implement convolution, recurrence, readout, normalization, and
snapshot writes. |
| `onnxruntime/test/contrib_ops/fused_state_space_op_test.cc,
onnxruntime/test/contrib_ops/cuda_kernels/fused_state_space_test.cc` |
Exercise provider-level state/selector behavior and internal CUDA
kernels. |
| `docs/contrib_ops/cuda/mamba2_verify.md, docs/ContribOperators.md,
docs/OperatorKernels.md` | Describe the operator ABI, snapshot behavior,
and CUDA support. |
## Testing
### Optional-output update - `3e40c0d4380572c4edd2aaf47b32883437bba418`
- `xbc` (output 3) and `readout` (output 4) are independently optional;
omitted
intermediates use stream-aware ORT scratch workspace. Required outputs
and all
existing output positions are unchanged.
- The scope remains one Bolt sequence per invocation. The public SSM
layout
remains `[slots, heads, d_state, head_dim]`. CUDA math, dtypes/cast
boundaries,
timestep clamping, state selection and the readout/replay guarantee are
unchanged.
- **PASS:** 19 repository gtests (5 schema, 10 CUDA provider, 4 internal
CUDA)
plus one fresh-kernel/schema guard: **20 checks, zero failures/skips**,
on H100.
Profiles record 66 `CUDAExecutionProvider/Mamba2Verify` node executions.
- **PASS:** all four intermediate-output combinations, including an
empty
output-3 placeholder; state aliases, lower/upper invalid selectors and
trailing/no-trailing snapshots. Eight capture cases (four combinations
at
one/eight rows) verify actual graph capture and bitwise replay.
- **PASS:** 5 unwrapped CPU-only schema tests, compiled without
`USE_CUDA` and
run with `CUDA_VISIBLE_DEVICES` empty; zero failures/skips.
- **PASS:** configured changed-range clang-format 20.1.8, whitespace
checks and
scoped official schema/kernel documentation rendering. The kernel-table
row
remains unchanged; generated schema documentation shows 3-5 outputs.
Qualification freshly compiles the operator host/device code, schema and
tests
against ABI-matched retained ORT 1.32.0 infrastructure. It is **not a
clean
full-source build, full-CI approval, independent HF parity or full-model
acceptance**.
The initial mixed-infrastructure-header harness failure was diagnosed in
`OpTester::BuildModel` and preserved; no production workaround or
threshold
relaxation was introduced. CUDA math/internal-test sources are
byte-unchanged.
### Historical validation before the optional-output update
**Review-fix snapshot:** `4f2bd30fe1f3017213ce9eb620b34ab507f5e392`.
Fixed static rank/dimension validation with CPU-only schema tests, dt
clamp/large-dt and NaN behavior, strict BF16 ULP/fraction checking and
nonuniform 1024-wide normalization. The slot-aware replay correction and
`MambaSingleRowReplayPreservesMultiRowReadout` regression are now owned
by this introducing PR (moved from #34003), with both host references
updated and the single-slot path preserved. The permanent contract is
now `docs/contrib_ops/cuda/mamba2_verify.md`.
**Owning-layer validation:** 8 provider + 3 CPU schema + 4 actual
internal CUDA gtests; zero failures/skips.
**Integrated tip regression:** 51 C++ provider/schema tests, 5 actual
internal CUDA tests and 2 Python IOBinding tests passed, zero
failures/skips. All owning-layer provider cases are retained. The final
overlay recompiles the changed CUDA/host/schema/test sources over
ABI-matched preserved ORT support; **not a clean full-source build,
full-CI approval, independent HF parity or new end-to-end
qualification**. Targeted configured clang-format/Ruff and whitespace
checks passed.
<details>
<summary>Historical source provenance moved out of permanent
documentation</summary>
Origin: #33940, commit `491dc043952066267b2212ba979f0325da318573`,
frozen external source `export-nemotron-pr-20261030/src/bolt_ops.cu`,
SHA256
`1f587d9eacc356805c37d19e94e7aa4465dd50302576b2ca4b9579d7c5ae8ba5`. This
identifies the original retained implementation, not a dependency in the
merged tree or a promise that later dispatch fixes are absent. The
slot-aware replay correction and current contract are documented in this
PR and the permanent operator documentation. This is the Bolt precision
contract, not independent HuggingFace model parity.
</details>
<details>
<summary>Historical measurements and qualification evidence (not
current-head acceptance)</summary>
These previously reported results are retained for provenance. They have
not been rerun on the rebased head above; earlier CI/build status and
benchmark results do not establish current-head CI, accuracy, or merge
readiness.
### Measured CUDA kernel latency - 2026-11-07
BF16 projection/convolution, FP32 SSM state; 128 heads, eight groups,
head dimension 64, state size 128, channels 10240, projection stride
18560, four-tap convolution. One fused R-row production launch is
compared with R serial single-row production calls with exact state
handoff; every timed invocation starts from the same immutable initial
state. This compares three fused kernels with 3*R serial kernels.
Frozen **Bolt** precision contract: FP32 gate/normalization/gamma
followed by BF16 output; this is not an acceptance test for the earlier
Lightning cast-before-gamma contract. Conv snapshots, SSM snapshots and
xBC are bitwise equal for all three cases in both runs. Y/readout pass
normalized RMSE <=0.03 and cosine >=0.999; exact raw metrics are below.
The one-row control is approximately unchanged; eight/sixteen-row fusion
reduces latency.
| Rows | Serial single-row calls us | Fused multi-row call us | Ratio |
|---|---|---|---|
| 1 | 8.91 | 8.91 | 1.00x |
| 8 | 82.98 | 22.17 | 3.74x |
| 16 | 167.01 | 41.41 | 4.03x |
H100 80GB HBM3, CUDA 12.8.93, driver 580.159.04, GCC 11.4.0. CUDA-event
median per dispatch; 64 dispatches per captured CUDA graph, 50 warmups,
three graph warmups, and 15 paired timed batches alternating AB/BA. Four
rotating allocation banks contain deterministic synthetic weights (seed
617); this is not a cold-cache guarantee or a checkpoint-weight
benchmark. Raw p95 values describe batch-average latency, not
individual-invocation tails. No profiler runs in the timing window. The
assigned GPU had empty admission, one stable compute client during
execution, and clean release afterward; clocks were not locked. Two
independent 39-case runs passed, including the published replay runner.
**Scope:** same-build production CUDA dispatch alternatives, not
historical parent-commit binaries. The measured runtime is an isolated
CUDA provider relink with preserved core/support, **not a clean
full-source ORT build or end-to-end model qualification**. Each measured
operator's CUDA source hash matches its owning PR head. A clean source
build is a separate replication variant and must be measured again.
Kernel-only timings do not establish model TTFT/tok/s, MMLU/LCB,
speculative integration, or full-model readiness.
[Complete primary/repeat results, numerical metrics, source hashes and
benchmark
code](https://github.com/microsoft/onnxruntime/pull/33969#issuecomment-6711210138).
All 39 cases passed in both supervised runs; full-model work remains
paused. No source push, stack change, merge, auto-merge or
ready-for-review promotion is included.
</details>
## Checklist
- [x] Review feedback addressed with focused regressions
- [x] Documentation and supported contracts updated
- [x] Native stack order and immediate-parent dependencies preserved
- [ ] Clean full-source/full-CI qualification (not claimed by focused
overlays)
---------
Signed-off-by: rui-ren <ruiren1225@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f4d0e1a3-a389-4cc8-9c85-0fd0e3ffce5b