[CUDA] Accelerate Nemotron Bolt large-chunk prefill across contrib ops (#34164)
## Description
This PR builds on #34147 (`main`) to enable and accelerate large-chunk
(`R` up to `4,096`) prefill across the native `com.microsoft` CUDA
contrib operators (`Mamba2Verify`, `MoEPacked`, `MoEGroupRouter`,
`MatMulPacked`, and `GroupQueryAttention`) for
`NVIDIA-Nemotron-3.5-Bolt-74B-A9B`. Combined with the companion
`onnxruntime-genai` dynamic-prefill runtime in
[microsoft/onnxruntime-genai#2961](https://github.com/microsoft/onnxruntime-genai/pull/2961),
prefill throughput on NVIDIA H200 (`sm_90`) reaches **`10,185 tok/s`
(`k=7`) / `10,429 tok/s` (`k=0`) at `L = 2048`** and **`11,424 tok/s`
(`k=7`) / `11,587 tok/s` (`k=0`) at `L = 3968`** — **`2.45×–2.67×`
faster than `llama.cpp` peak (`ub=4096`)** and **`13.98×–15.73×` faster
than fixed `R=8` prefill**.
## Summary of Changes
### `Mamba2Verify` Large-`R` Prefill & Right-Aligned Snapshots
| File | Change |
|------|--------|
|
`onnxruntime/core/graph/contrib_ops/fused_state_space_defs.h`<br>`onnxruntime/contrib_ops/cuda/sequence/fused_state_space.cc`<br>`docs/ContribOperators.md`
| Lift the `R <= 16` and `slots >= R` constraints (`R >= 1`, `slots >=
1`). When `R <= slots`, behavior is unchanged (`0..R-1` written,
`R..slots-1` preserved). When `R > slots`, only the trailing `slots`
tokens (`R - slots .. R - 1`) write snapshots `0 .. slots - 1` so slot
`slots - 1` holds the final recurrent state without writing intermediate
HBM snapshots. |
| `onnxruntime/contrib_ops/cuda/sequence/fused_state_space_impl.cu` |
Add 2D sequence-and-channel tiled `MambaConvPrefillKernel`, race-free
`MambaConvWriteSnapshotsKernel` (safe under in-place `conv_state`
aliasing), and `MambaSsmPrefillKernel<4, 16>` (register-resident state
scan with 36 KiB tiled shared-memory staging and exact 4-partition
readout parity). Aliasable state pointers are not `__restrict__`, and
the grouped RMS normalization launch is sliced into 65,535-row grid-y
chunks so any `R <= INT_MAX` launches. |
### `MoEPacked`, `MoEGroupRouter`, `MatMulPacked` &
`GroupQueryAttention` Prefill Optimizations
| File | Change |
|------|--------|
|
`onnxruntime/contrib_ops/cuda/moe/moe_packed.h`<br>`onnxruntime/contrib_ops/cuda/moe/moe_packed.cu`
| Raise `kMoEPackedMaxChunkRows` to `4096` so `L <= 4096` executes in a
single chunk, and add `BuildMoEPackedTilesKernel<64>` +
`MoEPackedReluIq2PrefillTcKernel<kFc1, 8>` so each IQ2_XXS weight
sub-block is decoded once per up-to-64 routes in shared memory instead
of once per 8 routes. Each MMA accumulator chain covers one 64-wide K
step and partials are combined with FP32 adds, matching the rounding of
the warp-split decode kernels (~1.4% kernel time on A100). |
|
`onnxruntime/contrib_ops/cuda/moe/moe_group_router_impl.h`<br>`onnxruntime/contrib_ops/cuda/moe/moe_group_router_impl.cu`<br>`onnxruntime/contrib_ops/cuda/bert/linear_attention_gates.cc`
| Add a zero-workspace `cublasGemmEx` Tensor Core GEMM + in-place
`SigmoidLogitsKernel` path for `rows > 16`, while keeping
`SigmoidScoresR8Kernel` for `rows == 8`. The schema now states that the
FP32 projection reduction order depends on the row count. |
| `onnxruntime/contrib_ops/cuda/quantization/matmul_packed.cu` | Add
templated `MatMulPackedBf16DequantFastKernel` (`IQ2_XXS`, `NVFP4`,
`Q8_0`), widen BF16 prefill tile width up to `18560` so projections
execute in a single `cublasGemmEx` call, and apply `NVFP4`
`global_scale` via `CUBLAS_POINTER_MODE_DEVICE` to eliminate
intermediate FP32 buffers and `M=128` sub-tiling. The cuBLAS epilogue
can round `global_scale` products one ulp differently from an FP32
multiply followed by the cast (documented in the schema). The activation
epilogue is sliced into 65,535-row grid-y chunks. |
|
`onnxruntime/contrib_ops/cuda/bert/group_query_attention.cc`<br>`onnxruntime/contrib_ops/cuda/bert/group_query_attention_impl.cu`<br>`onnxruntime/contrib_ops/cuda/bert/attention_data.h`<br>`onnxruntime/contrib_ops/cuda/bert/cudnn_fmha/cudnn_flash_attention.h`<br>`onnxruntime/contrib_ops/cuda/bert/cudnn_fmha/cudnn_flash_attention.cc`
| Prefer `FlashAttention` for small `R <= 16` shared-KV verify steps in
auto mode (so `R=8` verify uses `FlashAttention` without
`ORT_ENABLE_CUDNN_FLASH_ATTENTION=0` while large-`R` prefill uses Hopper
`cuDNN SDPA`), keep `cudnn_sdpa` during CUDA graph capture when its plan
is already cached (`cudnn_sdpa::has_cached_graph`) and fall back only
when a non-capturable plan build would be needed, and route cuDNN SDPA
workspace allocations through arena-pooled `GetScratchBuffer`
(eliminating synchronous `cudaMalloc`/`cudaFree` stalls). |
### Follow-ups from #34147
| File | Change |
|------|--------|
| `onnxruntime/contrib_ops/cuda/moe/moe_packed.cu` | Use zero
coefficients when `sum + routing_epsilon == 0` (previously Inf/NaN
rows), matching the documented normalization contract. |
|
`onnxruntime/contrib_ops/cuda/moe/moe_packed.cc`<br>`onnxruntime/core/graph/contrib_ops/contrib_defs.cc`
| Reject non-default `routed_scaling_factor`/`routing_epsilon` without
`router_weights`; legacy softmax routing silently ignored them. |
|
`onnxruntime/core/graph/contrib_ops/bert_defs.cc`<br>`docs/ContribOperators.md`
| Document the row-count-dependent `MoEGroupRouter` projection reduction
order (8-row split-K kernel, cuBLAS for `rows > 16`). |
### Unit Tests
| File | Change |
|------|--------|
| `onnxruntime/test/contrib_ops/fused_state_space_op_test.cc` | Update
schema validation test for `slots >= 1`. |
| `onnxruntime/test/contrib_ops/cuda_kernels/fused_state_space_test.cc`
| Add `MambaNormRowsBeyondGridYLimit` (65,537 rows) and
`MambaPrefillInPlaceStatesMatchSeparateOutputs` (aliased vs separate
state buffers for `R = 20/37`, including `state_row = 0`). |
| `onnxruntime/test/contrib_ops/moe_packed_routing_test.cc` | Add
legacy-routing rejection cases for `routed_scaling_factor` and
`routing_epsilon`. |
| `onnxruntime/test/contrib_ops/matmul_packed_precision_test.cc` | Allow
one ulp for BF16 NVFP4 `global_scale` prefill rows (decode and FP32
prefill stay bit-exact). |
| `onnxruntime/test/python/transformers/test_gqa.py` | Add
`test_cuda_graph_capture_reuses_warm_cudnn_sdpa_plan`. |
| `onnxruntime/test/python/transformers/test_gguf_packed_ops.py` | Add
`test_mamba2_verify_prefill_right_aligned_snapshots` (`R = 37, slots =
8` vs sequential `R = 8` chunks) and `m = 65` Tensor Core GEMM coverage
in `test_moe_group_router_r8_and_fused_bias`; add
`test_mamba2_verify_prefill_rows_beyond_grid_y_limit`,
`test_moe_packed_routing_epsilon_zero_denominator` and
`test_matmul_packed_activation_rows_beyond_grid_y_limit`. |
## Motivation and Context
### Prefill Throughput on NVIDIA H200 (`sm_90`, CUDA 13.0,
`NVIDIA-Nemotron-3.5-Bolt-74B-A9B`)
| Prompt `L` | `llama.cpp` Default (`ub=512`) | `llama.cpp` Peak
(`ub=4096`) | Prior ORT (`R=8`, `k=7`) | **This PR (`k=0`)** | **This PR
(`k=7`)** | **Speedup (`k=7`) vs `llama.cpp` Peak / vs Prior ORT** |
|---:|---:|---:|---:|---:|---:|---:|
| **128** | 85.96 ms (`1,489.0 tok/s`) | 86.20 ms (`1,484.9 tok/s`) |
175.28 ms (`730.3 tok/s`) | **52.80 ms (`2,424.3 tok/s`)** | **55.66 ms
(`2,299.6 tok/s`)** | **`1.55×`** / **`3.15×`** |
| **512** | 166.26 ms (`3,079.5 tok/s`) | 166.86 ms (`3,068.4 tok/s`) |
700.32 ms (`731.1 tok/s`) | **83.82 ms (`6,108.6 tok/s`)** | **87.66 ms
(`5,840.7 tok/s`)** | **`1.90×`** / **`7.99×`** |
| **1024** | 332.31 ms (`3,081.4 tok/s`) | 271.43 ms (`3,772.6 tok/s`) |
1,391.56 ms (`735.9 tok/s`) | **120.56 ms (`8,493.6 tok/s`)** | **125.06
ms (`8,187.8 tok/s`)** | **`2.17×`** / **`11.13×`** |
| **2048** | 671.54 ms (`3,049.7 tok/s`) | 493.12 ms (`4,153.2 tok/s`) |
2,810.28 ms (`728.8 tok/s`) | **196.39 ms (`10,428.5 tok/s`)** |
**201.08 ms (`10,184.8 tok/s`)** | **`2.45×`** / **`13.98×`** |
| **3968** | 1,320.22 ms (`3,005.6 tok/s`) | 915.41 ms (`4,334.7 tok/s`)
| 5,462.94 ms (`726.3 tok/s`) | **342.46 ms (`11,586.9 tok/s`)** |
**347.34 ms (`11,423.8 tok/s`)** | **`2.64×`** / **`15.73×`** |
### `nsys` Target Kernel-Family Breakdown at `L = 2048` (`192.76 ms` vs
`483.81 ms` in `llama.cpp` = **`2.51×` faster**)
- **IQ2_XXS MoE Routed GEMMs** (`40` layers): `98.51 ms` vs `137.61 ms`
(**`1.40×` faster**)
- **Mamba2 Conv / SSM Scan / Norm** (`40` layers): `46.42 ms` vs `47.82
ms` (**`1.03×` faster**; conv `2.43 ms` vs `3.42 ms`)
- **NVFP4 Mamba2 Projections** (`40` layers): `25.69 ms` vs `123.21 ms`
(**`4.80×` faster**)
- **IQ2_XXS MoE Shared GEMMs** (`40` layers): `11.63 ms` vs `35.56 ms`
(**`3.06×` faster**)
- **Q8_0 Projections + `M=1` LM Head**: `4.64 ms` vs `13.22 ms`
(**`2.85×` faster**)
- **MoE Routing / Tile Build / Finalize**: `2.39 ms` vs `77.68 ms`
(**`32.45×` faster**)
- **GQA Attention + RoPE / KV**: `1.08 ms` vs `2.76 ms` (**`2.55×`
faster**, `0` synchronous `cudaMalloc`/`cudaFree` calls)
## Testing
- Ran `onnxruntime/test/python/transformers/test_gguf_packed_ops.py` on
H200 (`sm_90`), including
`test_mamba2_verify_prefill_right_aligned_snapshots`,
`test_moe_group_router_r8_and_fused_bias`,
`test_moe_packed_routed_scaling_factor_and_routing_epsilon`, and
`test_matmul_packed_activation_types`.
- Verified end-to-end MMLU-Pro 200 accuracy (`119 / 200 = 59.50%`,
`97.0%` exact choice agreement with #34147) and speculative decode
throughput (`k=7`) on `NVIDIA-Nemotron-3.5-Bolt-74B-A9B`.
- Verified formatting and linting with `lintrunner -a` (`No lint
issues`).
- On A100 (`sm_80`, CUDA 13.0): `test_gguf_packed_ops.py` (50 tests),
`onnxruntime_provider_test
--gtest_filter='MoEPacked*:MatMulPacked*:MoEGroupRouter*:FusedStateSpace*:Mamba*:GatedDeltaNet*:GroupQueryAttention*:*MultiHeadAttention*:AttentionTest*:PagedAttention*'`
(478 tests), `FusedStateSpaceKernelTest.*` via `CUDA_EP_Unittest.All`,
and the `test_gqa.py` cuDNN/CUDA graph subset. The new boundary,
zero-denominator and CUDA graph tests fail on dc36d1465d.
## Checklist
- [x] Tests added/updated
- [x] No breaking changes (backward-compatible with `1 <= R <= slots`)
- [x] Documentation updated (`docs/ContribOperators.md`)
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>