onnxruntime
d2f77cd9 - [CUDA] Accelerate Nemotron Bolt large-chunk prefill across contrib ops (#34164)

Commit
4 days ago
[CUDA] Accelerate Nemotron Bolt large-chunk prefill across contrib ops (#34164) ## Description This PR builds on #34147 (`main`) to enable and accelerate large-chunk (`R` up to `4,096`) prefill across the native `com.microsoft` CUDA contrib operators (`Mamba2Verify`, `MoEPacked`, `MoEGroupRouter`, `MatMulPacked`, and `GroupQueryAttention`) for `NVIDIA-Nemotron-3.5-Bolt-74B-A9B`. Combined with the companion `onnxruntime-genai` dynamic-prefill runtime in [microsoft/onnxruntime-genai#2961](https://github.com/microsoft/onnxruntime-genai/pull/2961), prefill throughput on NVIDIA H200 (`sm_90`) reaches **`10,185 tok/s` (`k=7`) / `10,429 tok/s` (`k=0`) at `L = 2048`** and **`11,424 tok/s` (`k=7`) / `11,587 tok/s` (`k=0`) at `L = 3968`** — **`2.45×–2.67×` faster than `llama.cpp` peak (`ub=4096`)** and **`13.98×–15.73×` faster than fixed `R=8` prefill**. ## Summary of Changes ### `Mamba2Verify` Large-`R` Prefill & Right-Aligned Snapshots | File | Change | |------|--------| | `onnxruntime/core/graph/contrib_ops/fused_state_space_defs.h`<br>`onnxruntime/contrib_ops/cuda/sequence/fused_state_space.cc`<br>`docs/ContribOperators.md` | Lift the `R <= 16` and `slots >= R` constraints (`R >= 1`, `slots >= 1`). When `R <= slots`, behavior is unchanged (`0..R-1` written, `R..slots-1` preserved). When `R > slots`, only the trailing `slots` tokens (`R - slots .. R - 1`) write snapshots `0 .. slots - 1` so slot `slots - 1` holds the final recurrent state without writing intermediate HBM snapshots. | | `onnxruntime/contrib_ops/cuda/sequence/fused_state_space_impl.cu` | Add 2D sequence-and-channel tiled `MambaConvPrefillKernel`, race-free `MambaConvWriteSnapshotsKernel` (safe under in-place `conv_state` aliasing), and `MambaSsmPrefillKernel<4, 16>` (register-resident state scan with 36 KiB tiled shared-memory staging and exact 4-partition readout parity). Aliasable state pointers are not `__restrict__`, and the grouped RMS normalization launch is sliced into 65,535-row grid-y chunks so any `R <= INT_MAX` launches. | ### `MoEPacked`, `MoEGroupRouter`, `MatMulPacked` & `GroupQueryAttention` Prefill Optimizations | File | Change | |------|--------| | `onnxruntime/contrib_ops/cuda/moe/moe_packed.h`<br>`onnxruntime/contrib_ops/cuda/moe/moe_packed.cu` | Raise `kMoEPackedMaxChunkRows` to `4096` so `L <= 4096` executes in a single chunk, and add `BuildMoEPackedTilesKernel<64>` + `MoEPackedReluIq2PrefillTcKernel<kFc1, 8>` so each IQ2_XXS weight sub-block is decoded once per up-to-64 routes in shared memory instead of once per 8 routes. Each MMA accumulator chain covers one 64-wide K step and partials are combined with FP32 adds, matching the rounding of the warp-split decode kernels (~1.4% kernel time on A100). | | `onnxruntime/contrib_ops/cuda/moe/moe_group_router_impl.h`<br>`onnxruntime/contrib_ops/cuda/moe/moe_group_router_impl.cu`<br>`onnxruntime/contrib_ops/cuda/bert/linear_attention_gates.cc` | Add a zero-workspace `cublasGemmEx` Tensor Core GEMM + in-place `SigmoidLogitsKernel` path for `rows > 16`, while keeping `SigmoidScoresR8Kernel` for `rows == 8`. The schema now states that the FP32 projection reduction order depends on the row count. | | `onnxruntime/contrib_ops/cuda/quantization/matmul_packed.cu` | Add templated `MatMulPackedBf16DequantFastKernel` (`IQ2_XXS`, `NVFP4`, `Q8_0`), widen BF16 prefill tile width up to `18560` so projections execute in a single `cublasGemmEx` call, and apply `NVFP4` `global_scale` via `CUBLAS_POINTER_MODE_DEVICE` to eliminate intermediate FP32 buffers and `M=128` sub-tiling. The cuBLAS epilogue can round `global_scale` products one ulp differently from an FP32 multiply followed by the cast (documented in the schema). The activation epilogue is sliced into 65,535-row grid-y chunks. | | `onnxruntime/contrib_ops/cuda/bert/group_query_attention.cc`<br>`onnxruntime/contrib_ops/cuda/bert/group_query_attention_impl.cu`<br>`onnxruntime/contrib_ops/cuda/bert/attention_data.h`<br>`onnxruntime/contrib_ops/cuda/bert/cudnn_fmha/cudnn_flash_attention.h`<br>`onnxruntime/contrib_ops/cuda/bert/cudnn_fmha/cudnn_flash_attention.cc` | Prefer `FlashAttention` for small `R <= 16` shared-KV verify steps in auto mode (so `R=8` verify uses `FlashAttention` without `ORT_ENABLE_CUDNN_FLASH_ATTENTION=0` while large-`R` prefill uses Hopper `cuDNN SDPA`), keep `cudnn_sdpa` during CUDA graph capture when its plan is already cached (`cudnn_sdpa::has_cached_graph`) and fall back only when a non-capturable plan build would be needed, and route cuDNN SDPA workspace allocations through arena-pooled `GetScratchBuffer` (eliminating synchronous `cudaMalloc`/`cudaFree` stalls). | ### Follow-ups from #34147 | File | Change | |------|--------| | `onnxruntime/contrib_ops/cuda/moe/moe_packed.cu` | Use zero coefficients when `sum + routing_epsilon == 0` (previously Inf/NaN rows), matching the documented normalization contract. | | `onnxruntime/contrib_ops/cuda/moe/moe_packed.cc`<br>`onnxruntime/core/graph/contrib_ops/contrib_defs.cc` | Reject non-default `routed_scaling_factor`/`routing_epsilon` without `router_weights`; legacy softmax routing silently ignored them. | | `onnxruntime/core/graph/contrib_ops/bert_defs.cc`<br>`docs/ContribOperators.md` | Document the row-count-dependent `MoEGroupRouter` projection reduction order (8-row split-K kernel, cuBLAS for `rows > 16`). | ### Unit Tests | File | Change | |------|--------| | `onnxruntime/test/contrib_ops/fused_state_space_op_test.cc` | Update schema validation test for `slots >= 1`. | | `onnxruntime/test/contrib_ops/cuda_kernels/fused_state_space_test.cc` | Add `MambaNormRowsBeyondGridYLimit` (65,537 rows) and `MambaPrefillInPlaceStatesMatchSeparateOutputs` (aliased vs separate state buffers for `R = 20/37`, including `state_row = 0`). | | `onnxruntime/test/contrib_ops/moe_packed_routing_test.cc` | Add legacy-routing rejection cases for `routed_scaling_factor` and `routing_epsilon`. | | `onnxruntime/test/contrib_ops/matmul_packed_precision_test.cc` | Allow one ulp for BF16 NVFP4 `global_scale` prefill rows (decode and FP32 prefill stay bit-exact). | | `onnxruntime/test/python/transformers/test_gqa.py` | Add `test_cuda_graph_capture_reuses_warm_cudnn_sdpa_plan`. | | `onnxruntime/test/python/transformers/test_gguf_packed_ops.py` | Add `test_mamba2_verify_prefill_right_aligned_snapshots` (`R = 37, slots = 8` vs sequential `R = 8` chunks) and `m = 65` Tensor Core GEMM coverage in `test_moe_group_router_r8_and_fused_bias`; add `test_mamba2_verify_prefill_rows_beyond_grid_y_limit`, `test_moe_packed_routing_epsilon_zero_denominator` and `test_matmul_packed_activation_rows_beyond_grid_y_limit`. | ## Motivation and Context ### Prefill Throughput on NVIDIA H200 (`sm_90`, CUDA 13.0, `NVIDIA-Nemotron-3.5-Bolt-74B-A9B`) | Prompt `L` | `llama.cpp` Default (`ub=512`) | `llama.cpp` Peak (`ub=4096`) | Prior ORT (`R=8`, `k=7`) | **This PR (`k=0`)** | **This PR (`k=7`)** | **Speedup (`k=7`) vs `llama.cpp` Peak / vs Prior ORT** | |---:|---:|---:|---:|---:|---:|---:| | **128** | 85.96 ms (`1,489.0 tok/s`) | 86.20 ms (`1,484.9 tok/s`) | 175.28 ms (`730.3 tok/s`) | **52.80 ms (`2,424.3 tok/s`)** | **55.66 ms (`2,299.6 tok/s`)** | **`1.55×`** / **`3.15×`** | | **512** | 166.26 ms (`3,079.5 tok/s`) | 166.86 ms (`3,068.4 tok/s`) | 700.32 ms (`731.1 tok/s`) | **83.82 ms (`6,108.6 tok/s`)** | **87.66 ms (`5,840.7 tok/s`)** | **`1.90×`** / **`7.99×`** | | **1024** | 332.31 ms (`3,081.4 tok/s`) | 271.43 ms (`3,772.6 tok/s`) | 1,391.56 ms (`735.9 tok/s`) | **120.56 ms (`8,493.6 tok/s`)** | **125.06 ms (`8,187.8 tok/s`)** | **`2.17×`** / **`11.13×`** | | **2048** | 671.54 ms (`3,049.7 tok/s`) | 493.12 ms (`4,153.2 tok/s`) | 2,810.28 ms (`728.8 tok/s`) | **196.39 ms (`10,428.5 tok/s`)** | **201.08 ms (`10,184.8 tok/s`)** | **`2.45×`** / **`13.98×`** | | **3968** | 1,320.22 ms (`3,005.6 tok/s`) | 915.41 ms (`4,334.7 tok/s`) | 5,462.94 ms (`726.3 tok/s`) | **342.46 ms (`11,586.9 tok/s`)** | **347.34 ms (`11,423.8 tok/s`)** | **`2.64×`** / **`15.73×`** | ### `nsys` Target Kernel-Family Breakdown at `L = 2048` (`192.76 ms` vs `483.81 ms` in `llama.cpp` = **`2.51×` faster**) - **IQ2_XXS MoE Routed GEMMs** (`40` layers): `98.51 ms` vs `137.61 ms` (**`1.40×` faster**) - **Mamba2 Conv / SSM Scan / Norm** (`40` layers): `46.42 ms` vs `47.82 ms` (**`1.03×` faster**; conv `2.43 ms` vs `3.42 ms`) - **NVFP4 Mamba2 Projections** (`40` layers): `25.69 ms` vs `123.21 ms` (**`4.80×` faster**) - **IQ2_XXS MoE Shared GEMMs** (`40` layers): `11.63 ms` vs `35.56 ms` (**`3.06×` faster**) - **Q8_0 Projections + `M=1` LM Head**: `4.64 ms` vs `13.22 ms` (**`2.85×` faster**) - **MoE Routing / Tile Build / Finalize**: `2.39 ms` vs `77.68 ms` (**`32.45×` faster**) - **GQA Attention + RoPE / KV**: `1.08 ms` vs `2.76 ms` (**`2.55×` faster**, `0` synchronous `cudaMalloc`/`cudaFree` calls) ## Testing - Ran `onnxruntime/test/python/transformers/test_gguf_packed_ops.py` on H200 (`sm_90`), including `test_mamba2_verify_prefill_right_aligned_snapshots`, `test_moe_group_router_r8_and_fused_bias`, `test_moe_packed_routed_scaling_factor_and_routing_epsilon`, and `test_matmul_packed_activation_types`. - Verified end-to-end MMLU-Pro 200 accuracy (`119 / 200 = 59.50%`, `97.0%` exact choice agreement with #34147) and speculative decode throughput (`k=7`) on `NVIDIA-Nemotron-3.5-Bolt-74B-A9B`. - Verified formatting and linting with `lintrunner -a` (`No lint issues`). - On A100 (`sm_80`, CUDA 13.0): `test_gguf_packed_ops.py` (50 tests), `onnxruntime_provider_test --gtest_filter='MoEPacked*:MatMulPacked*:MoEGroupRouter*:FusedStateSpace*:Mamba*:GatedDeltaNet*:GroupQueryAttention*:*MultiHeadAttention*:AttentionTest*:PagedAttention*'` (478 tests), `FusedStateSpaceKernelTest.*` via `CUDA_EP_Unittest.All`, and the `test_gqa.py` cuDNN/CUDA graph subset. The new boundary, zero-denominator and CUDA graph tests fail on dc36d1465d. ## Checklist - [x] Tests added/updated - [x] No breaking changes (backward-compatible with `1 <= R <= slots`) - [x] Documentation updated (`docs/ContribOperators.md`) --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Author
Parents
Loading