[CUDA] Optimize MoEPacked, MoEGroupRouter, and MatMulPacked for Nemotron Bolt (#34147)
## Description
This PR stacks on top of #34003 to close the remaining prefill and
decode latency gap between native `com.microsoft` contrib ops
(`MoEGroupRouter`, `MoEPacked`, `MatMulPacked`) and the `v11_gguf`
custom-op baseline on the 88-layer `NVIDIA-Nemotron-3.5-Bolt-74B-A9B`
model (`rc7`). It extends `MoEPacked` and `MatMulPacked` with optional
fusion attributes (`routed_scaling_factor`, `routing_epsilon`,
`activation_type`) so per-layer Top-K routing normalization/scaling and
shared-expert ReLU² epilogues fuse directly into the operator kernels,
and optimizes the CUDA `MoEGroupRouter`, `MoEPacked`, and `MatMulPacked`
kernels on Hopper (`sm_90`).
## Summary of Changes
### Operator Schema & Documentation
| File | Change |
|------|--------|
| `onnxruntime/core/graph/contrib_ops/contrib_defs.cc` | Add
`routed_scaling_factor` (`FLOAT`, default `1.0`) and `routing_epsilon`
(`FLOAT`, default `0.0`) to `com.microsoft::MoEPacked(1)`, and
`activation_type` (`STRING`, default `"none"`, supporting `"none"`,
`"relu2"`/`"relu_squared"`, `"silu"`) to
`com.microsoft::MatMulPacked(1)`. |
| `docs/ContribOperators.md` | Update generated contrib operator schema
documentation for `MatMulPacked` and `MoEPacked`. |
### CUDA Kernel Optimizations
| File | Change |
|------|--------|
| `onnxruntime/contrib_ops/cuda/moe/moe_group_router_impl.cu` | Fuse
`AddSelectionBiasKernel` into `SigmoidScoresKernel`,
`SigmoidScoresR8Kernel`, and `SigmoidLogitsKernel` (eliminating 40
separate kernel launches per step); add `SigmoidScoresR8Kernel<CudaT,
kAligned>` (`<<<experts, 128>>>`, 4 warps splitting `K`) so `rows == 8`
speculative verify reads `projection_weight[e, :]` once for all 8 rows
(`11.89 us -> 5.76 us/layer`). |
|
`onnxruntime/contrib_ops/cuda/moe/moe_packed.h`<br>`onnxruntime/contrib_ops/cuda/moe/moe_packed.cc`<br>`onnxruntime/contrib_ops/cuda/moe/moe_packed.cu`
| Fuse `(selected_weight / (sum + routing_epsilon)) *
routed_scaling_factor` into `FullScoreRoutingKernel` and
`FullScoreRoutingGenericKernel`; launch 1 warp/CTA (`<<<rows, 32>>>`)
when `rows <= 64` (`18.18 us -> 9.81 us/layer`); hoist weight row
pointers and use 2-byte aligned `LoadIq2SubBlock16Stage` +
`DecodeIq2StageToBf16` with `__launch_bounds__(KW * 32, ...)` in
`MoEPackedReluIq2ContiguousKKernel` across all `K` (including `K=1280`
5-block FC2). |
| `onnxruntime/contrib_ops/cuda/quantization/gguf_quant_cuda.cuh` | Add
`LoadIq2SubBlock16Stage<kAlignedU16>` and `DecodeIq2StageToBf16` (using
`bf16x2` sign-mask bit injection before FP32 scale multiplication and
single `bfloat162` rounding) and `LoadQ8SubBlock16AlignedStage` +
`DecodeQ8StageToBf16` to eliminate intermediate float arrays and reduce
register pressure. |
|
`onnxruntime/contrib_ops/cuda/quantization/matmul_packed.h`<br>`onnxruntime/contrib_ops/cuda/quantization/matmul_packed.cc`<br>`onnxruntime/contrib_ops/cuda/quantization/matmul_packed.cu`
| Add `activation_type` epilogue fusion (`ApplyMatMulPackedActivation`)
across Tensor Core, GEMV, and prefill paths, and use
`LoadIq2SubBlock16Stage`/`DecodeIq2StageToBf16` and
`LoadQ8SubBlock16AlignedStage`/`DecodeQ8StageToBf16` (`Q8_0` LM head
`310.8 us -> 266.3 us`, `IQ2_XXS` `sh_dn` `19.30 us -> 14.77 us`). |
### Unit Tests
| File | Change |
|------|--------|
| `onnxruntime/test/python/transformers/test_gguf_packed_ops.py` | Add
`test_matmul_packed_activation_types`,
`test_moe_packed_routed_scaling_factor_and_routing_epsilon` (including
`inter_size=1280` 5-block IQ2_XXS), and
`test_moe_group_router_r8_and_fused_bias`. |
| `onnxruntime/test/contrib_ops/matmul_packed_precision_test.cc` | Add
`ActivationTypeRelu2AndSiluAcrossDispatchPaths`. |
| `onnxruntime/test/contrib_ops/moe_packed_routing_test.cc` | Add
validation tests for nonfinite `routed_scaling_factor` and
negative/nonfinite `routing_epsilon`. |
## Motivation and Context
On NVIDIA H200 (`sm_90`, CUDA 13.0) with
`NVIDIA-Nemotron-3.5-Bolt-74B-A9B` (`rc7`, 88 layers, `k=7` MTP verify
`rows=8`, CUDA graph enabled):
| Metric | `rc7` Baseline (PR #34003) | `rc7` + This PR | `v11_gguf`
Custom Ops | Delta vs PR #34003 |
|---|---:|---:|---:|---:|
| **ONNX Graph Nodes** | 2,912 | **1,272** | 1,155 | **-1,640 nodes
(-56.3%)** |
| **CUDA Graph Kernel + Memcpy Launches / Round** | 2,009 | **729** |
690 | **-1,280 launches (-63.7%)** |
| **`nsys` CUDA Graph Replay GPU Time / Round** | 16.562 ms | **12.408
ms** | 11.596 ms | **-4.154 ms (-25.1%)** |
| **MMLU-Pro 200 Accuracy (`k=7`)** | 119/200 (59.50%) | **121/200
(60.50%)** | 119/200 (59.50%) | **+1.00%** |
| **Prefill Throughput (`500` prompts, `k=7`)** | 536.5 tok/s | **729.3
tok/s** | 685.6 tok/s | **+35.9% (+6.4% vs `v11_gguf`)** |
| **Decode Throughput (`k=7`, normal EOS)** | 268.88 tok/s | **349.91
tok/s** | 378.89 tok/s | **+30.1%** |
| **Decode Throughput (`k=7`, `--disable-eos`)** | 304.67 tok/s |
**394.65 tok/s** | 428.88 tok/s | **+29.5%** |
| **Decode Throughput (`k=0`, no MTP)** | 77.60 tok/s | **106.10 tok/s**
| 109.76 tok/s | **+36.7% (96.7% of `v11_gguf`)** |
## Testing
- Ran `onnxruntime/test/python/transformers/test_gguf_packed_ops.py` on
H200 (`sm_90`), including all existing and newly added `MatMulPacked`,
`MoEPacked`, and `MoEGroupRouter` tests.
- Validated MMLU-Pro 200 accuracy (`121/200 = 60.50%`), prefill (`500`
prompts), and decode (`100` prompts, `k=7` and `k=0`) on
`NVIDIA-Nemotron-3.5-Bolt-74B-A9B` (`rc7`).
## Checklist
- [x] Tests added/updated
- [x] No breaking changes (all new attributes have backward-compatible
defaults)
- [x] Documentation updated (`docs/ContribOperators.md`)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>