onnxruntime
d5704db9 - [CUDA] Optimize MoEPacked, MoEGroupRouter, and MatMulPacked for Nemotron Bolt (#34147)

Commit
3 days ago
[CUDA] Optimize MoEPacked, MoEGroupRouter, and MatMulPacked for Nemotron Bolt (#34147) ## Description This PR stacks on top of #34003 to close the remaining prefill and decode latency gap between native `com.microsoft` contrib ops (`MoEGroupRouter`, `MoEPacked`, `MatMulPacked`) and the `v11_gguf` custom-op baseline on the 88-layer `NVIDIA-Nemotron-3.5-Bolt-74B-A9B` model (`rc7`). It extends `MoEPacked` and `MatMulPacked` with optional fusion attributes (`routed_scaling_factor`, `routing_epsilon`, `activation_type`) so per-layer Top-K routing normalization/scaling and shared-expert ReLU² epilogues fuse directly into the operator kernels, and optimizes the CUDA `MoEGroupRouter`, `MoEPacked`, and `MatMulPacked` kernels on Hopper (`sm_90`). ## Summary of Changes ### Operator Schema & Documentation | File | Change | |------|--------| | `onnxruntime/core/graph/contrib_ops/contrib_defs.cc` | Add `routed_scaling_factor` (`FLOAT`, default `1.0`) and `routing_epsilon` (`FLOAT`, default `0.0`) to `com.microsoft::MoEPacked(1)`, and `activation_type` (`STRING`, default `"none"`, supporting `"none"`, `"relu2"`/`"relu_squared"`, `"silu"`) to `com.microsoft::MatMulPacked(1)`. | | `docs/ContribOperators.md` | Update generated contrib operator schema documentation for `MatMulPacked` and `MoEPacked`. | ### CUDA Kernel Optimizations | File | Change | |------|--------| | `onnxruntime/contrib_ops/cuda/moe/moe_group_router_impl.cu` | Fuse `AddSelectionBiasKernel` into `SigmoidScoresKernel`, `SigmoidScoresR8Kernel`, and `SigmoidLogitsKernel` (eliminating 40 separate kernel launches per step); add `SigmoidScoresR8Kernel<CudaT, kAligned>` (`<<<experts, 128>>>`, 4 warps splitting `K`) so `rows == 8` speculative verify reads `projection_weight[e, :]` once for all 8 rows (`11.89 us -> 5.76 us/layer`). | | `onnxruntime/contrib_ops/cuda/moe/moe_packed.h`<br>`onnxruntime/contrib_ops/cuda/moe/moe_packed.cc`<br>`onnxruntime/contrib_ops/cuda/moe/moe_packed.cu` | Fuse `(selected_weight / (sum + routing_epsilon)) * routed_scaling_factor` into `FullScoreRoutingKernel` and `FullScoreRoutingGenericKernel`; launch 1 warp/CTA (`<<<rows, 32>>>`) when `rows <= 64` (`18.18 us -> 9.81 us/layer`); hoist weight row pointers and use 2-byte aligned `LoadIq2SubBlock16Stage` + `DecodeIq2StageToBf16` with `__launch_bounds__(KW * 32, ...)` in `MoEPackedReluIq2ContiguousKKernel` across all `K` (including `K=1280` 5-block FC2). | | `onnxruntime/contrib_ops/cuda/quantization/gguf_quant_cuda.cuh` | Add `LoadIq2SubBlock16Stage<kAlignedU16>` and `DecodeIq2StageToBf16` (using `bf16x2` sign-mask bit injection before FP32 scale multiplication and single `bfloat162` rounding) and `LoadQ8SubBlock16AlignedStage` + `DecodeQ8StageToBf16` to eliminate intermediate float arrays and reduce register pressure. | | `onnxruntime/contrib_ops/cuda/quantization/matmul_packed.h`<br>`onnxruntime/contrib_ops/cuda/quantization/matmul_packed.cc`<br>`onnxruntime/contrib_ops/cuda/quantization/matmul_packed.cu` | Add `activation_type` epilogue fusion (`ApplyMatMulPackedActivation`) across Tensor Core, GEMV, and prefill paths, and use `LoadIq2SubBlock16Stage`/`DecodeIq2StageToBf16` and `LoadQ8SubBlock16AlignedStage`/`DecodeQ8StageToBf16` (`Q8_0` LM head `310.8 us -> 266.3 us`, `IQ2_XXS` `sh_dn` `19.30 us -> 14.77 us`). | ### Unit Tests | File | Change | |------|--------| | `onnxruntime/test/python/transformers/test_gguf_packed_ops.py` | Add `test_matmul_packed_activation_types`, `test_moe_packed_routed_scaling_factor_and_routing_epsilon` (including `inter_size=1280` 5-block IQ2_XXS), and `test_moe_group_router_r8_and_fused_bias`. | | `onnxruntime/test/contrib_ops/matmul_packed_precision_test.cc` | Add `ActivationTypeRelu2AndSiluAcrossDispatchPaths`. | | `onnxruntime/test/contrib_ops/moe_packed_routing_test.cc` | Add validation tests for nonfinite `routed_scaling_factor` and negative/nonfinite `routing_epsilon`. | ## Motivation and Context On NVIDIA H200 (`sm_90`, CUDA 13.0) with `NVIDIA-Nemotron-3.5-Bolt-74B-A9B` (`rc7`, 88 layers, `k=7` MTP verify `rows=8`, CUDA graph enabled): | Metric | `rc7` Baseline (PR #34003) | `rc7` + This PR | `v11_gguf` Custom Ops | Delta vs PR #34003 | |---|---:|---:|---:|---:| | **ONNX Graph Nodes** | 2,912 | **1,272** | 1,155 | **-1,640 nodes (-56.3%)** | | **CUDA Graph Kernel + Memcpy Launches / Round** | 2,009 | **729** | 690 | **-1,280 launches (-63.7%)** | | **`nsys` CUDA Graph Replay GPU Time / Round** | 16.562 ms | **12.408 ms** | 11.596 ms | **-4.154 ms (-25.1%)** | | **MMLU-Pro 200 Accuracy (`k=7`)** | 119/200 (59.50%) | **121/200 (60.50%)** | 119/200 (59.50%) | **+1.00%** | | **Prefill Throughput (`500` prompts, `k=7`)** | 536.5 tok/s | **729.3 tok/s** | 685.6 tok/s | **+35.9% (+6.4% vs `v11_gguf`)** | | **Decode Throughput (`k=7`, normal EOS)** | 268.88 tok/s | **349.91 tok/s** | 378.89 tok/s | **+30.1%** | | **Decode Throughput (`k=7`, `--disable-eos`)** | 304.67 tok/s | **394.65 tok/s** | 428.88 tok/s | **+29.5%** | | **Decode Throughput (`k=0`, no MTP)** | 77.60 tok/s | **106.10 tok/s** | 109.76 tok/s | **+36.7% (96.7% of `v11_gguf`)** | ## Testing - Ran `onnxruntime/test/python/transformers/test_gguf_packed_ops.py` on H200 (`sm_90`), including all existing and newly added `MatMulPacked`, `MoEPacked`, and `MoEGroupRouter` tests. - Validated MMLU-Pro 200 accuracy (`121/200 = 60.50%`), prefill (`500` prompts), and decode (`100` prompts, `k=7` and `k=0`) on `NVIDIA-Nemotron-3.5-Bolt-74B-A9B` (`rc7`). ## Checklist - [x] Tests added/updated - [x] No breaking changes (all new attributes have backward-compatible defaults) - [x] Documentation updated (`docs/ContribOperators.md`) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Author
Parents
Loading