[CUDA] Optimize Q6_K/IQ3_XXS/IQ4_NL GGUF, FP16-KV DSA, M=8 HC, and FP32 GDN kernels (#34200)
## Summary
This PR optimizes CUDA prefill and speculative-decode (`M = 8` verify)
kernels across `MatMulPacked`, `MoEPacked`, `DynamicSparseAttention`
(`fp16_kv`), `HyperConnectionLowRankMix`, `GatedDeltaNet` (`fp32`
state), and `QMoE` helper launches for Qwen3.8-Flash-Next GGUF
(`UD-IQ3_XXS`) and `QMoE` (`2b375_fp32`) models:
1. **`MatMulPacked` `Q6_K` Contiguous Half-Superblock (`kChunk = 128`) &
Aligned SWAR Unpack** (`gguf_quant_cuda.cuh`, `matmul_packed.cu`):
- Assigns contiguous 128-element `Q6_K` half-superblocks per warp
(`LoadQ6KHalf128Stage` + `DecodeQ6KHalf128GroupBf16x2` SWAR unpack) in
`MatMulPackedBf16TcWarpKSplitKernel` (`KW=10, MT=1` for `N<=1024`;
`KW=8` for `K=6144, N<=2560`; `KW=4, MT=2` for `N>2560`) and enables
`Q6_K` in `MatMulPackedBf16DequantFastKernel`.
- Cuts `MatMulPacked` decode time from `19.60 -> 13.48 ms/cycle` (`-6.12
ms/cyc`; `N=512` `Q6_K` latency drops `5.5x` from `33.8 us -> ~6.1 us`).
2. **`MoEPacked` Contiguous `kChunk = 128` (`IQ3_XXS`, `IQ2_S`,
`IQ4_XS`) FC1 Decode & Aligned 36B Pair (`IQ4_NL`) FC2 Decode**
(`gguf_quant_cuda.cuh`, `moe_packed.cu`):
- Hoists superblock scale/sign loads across 4 sub-blocks per warp in
`MoEPackedFc1DecodeWarpKSplitKernel` with 4-way bank-replicated
`iq3xxs_grid` (`s_lut[1024]`), and decodes 4-byte-aligned 36B (`2 x
block_iq4_nl`) pairs (`DecodeIq4NlPair64AlignedBf16x2`) in
`MoEPackedFc2Bf16TcKernel<1, kGgmlTypeIQ4_NL>` (`21.47 -> 19.76 ms/cyc`,
reaching `295 GB/s` vs `301 GB/s` peak on RTX Spark).
3. **FP16-KV Tensor-Core `DynamicSparseAttentionFp16TcKernel<kSplitK,
kCandTile>`** (`dynamic_sparse_attention_prefill.cuh`,
`dynamic_sparse_attention_impl.cu`):
- Replaces the scalar `GroupedDecodeDynamicSparseAttentionKernel`
fallback on `fp16_kv` with a split-K `cp.async` Tensor-Core `m16n8k16`
FP16-KV kernel (`kCandTile = 32` on 100 KB/SM GPUs, `P_hi + P_residual`
22-bit probability split) for decode and two-buffer `<false, 64>` for
prefill.
- Speeds up FP16-KV attention decode by **`5.4x` (`2.71 -> 0.50
ms/cyc`)** and 8K prefill by **`1.49x` (`486.0 -> 325.3 ms`)**.
4. **Single-Wave `m16n8k16` Tensor-Core `HcDownTc8Kernel` &
`HcUpPreTc8Kernel` (`rows = 8`) + `NormVec8H2560Kernel`**
(`hyper_connection_low_rank_mix_impl.cu`):
- Replaces 2-wave scalar `LaunchFusedRows<T, 4, 2>` (`3.81 GB/cyc`
weight reads) on `rows=8, branches=4, hidden=2560, rank=320` (BF16) with
single-wave BF16 Tensor-Core kernels (`grid=(6,8,1)` and
`grid=(80,1,1)`) with PDL weight prefetch (`1.27 GB/cyc` = **`3.0x` less
weight DRAM traffic**, saving **`~3.5â4.5 ms/cyc`**), plus 320-thread
`NormVec8H2560Kernel` for prefill.
5. **Fused Dual-A 3-Slice FP32-State `GatedDeltaNetChunkedKernel`**
(`gated_delta_net_mma.cuh`, `gated_delta_net_impl.cu`):
- Hoists `LoadFragA` across slices and fuses `(k @ S0, Qg @ S0, R
epilogue)` (`SmemGemmFp32StateDualS0Bf16`) and `(P @ U, Kd^T @ U)`
(`SmemGemmFp32StateDualUBf16`), reducing 3-slice FP32->BF16 passes per
chunk from `5 -> 3` (`35 -> 12` barriers) while preserving exact 24-bit
IEEE-754 precision (`427.0 -> 343.3 ms` at 8K prefill, `-19.6%`).
6. **Column-Matched CTA Widths in `QMoE` `expandInputRowsKernel` &
`doActivationKernel`** (`moe_kernels.cu`,
`moe_gemm_activation_kernels.cuh`):
- Matches CTA block widths to column vector counts (`320` for
`hidden_size=2560`; `64`/`96` for `inter_size=512/640`), eliminating
69%â75% idle tail threads (`-31 ms` at 8K prefill).
---
## End-to-End Performance & Accuracy (`2Kâ255K + 1K` on NVIDIA RTX Spark
`SM 12.1`)
### 1. ORT `gguf_ud_iq3_xxs_q8_engram` (`DFlash2 K=7`) vs. Baseline &
`llama.cpp + MTP` (`UD-IQ3_XXS + Q8_0`)
| Context (`Prompt + Gen`) | ORT Baseline TTFT / Decode (`ms/cyc`) |
**ORT Optimized TTFT / Decode (`ms/cyc`)** | `llama.cpp + MTP` TTFT /
Decode | **ORT Opt vs. `llama.cpp + MTP`** |
|---|---:|---:|---:|---:|
| **`2K + 1K`** | `1.509 s` / `52.71 tok/s` (`67.62 ms`) | **`1.468 s` /
`66.70 tok/s` (`52.53 ms`)** | `2.582 s` / `57.81 tok/s` | **`1.76x`
prefill / `+15.4%` decode** |
| **`8K + 1K`** | `5.468 s` / `46.42 tok/s` (`67.81 ms`) | **`5.231 s` /
`66.49 tok/s` (`53.80 ms`)** | `9.616 s` / `54.22 tok/s` | **`1.84x`
prefill / `+22.6%` decode** |
| **`16K + 1K`** | `10.822 s` / `51.75 tok/s` (`67.24 ms`) | **`10.425
s` / `61.24 tok/s` (`54.06 ms`)** | `19.331 s` / `58.48 tok/s` |
**`1.85x` prefill / `+4.7%` decode** |
| **`32K + 1K`** | `21.663 s` / `48.44 tok/s` (`68.34 ms`) | **`20.929
s` / `63.54 tok/s` (`54.76 ms`)** | `40.822 s` / `54.69 tok/s` |
**`1.95x` prefill / `+16.2%` decode** |
| **`64K + 1K`** | `44.858 s` / `53.26 tok/s` (`70.09 ms`) | **`42.684
s` / `58.04 tok/s` (`57.23 ms`)** | `81.592 s` / `50.67 tok/s` |
**`1.91x` prefill / `+14.5%` decode** |
| **`128K + 1K`** | `94.444 s` / `42.02 tok/s` (`72.02 ms`) | **`91.288
s` / `59.12 tok/s` (`58.06 ms`)** | `172.053 s` / `45.89 tok/s` |
**`1.88x` prefill / `+28.8%` decode** |
| **`255K + 1K`** | `206.564 s` / `42.15 tok/s` (`81.71 ms`) |
**`201.523 s` / `45.41 tok/s` (`67.65 ms`)** | `378.801 s` / `33.42
tok/s` | **`1.88x` prefill / `+35.9%` decode** |
### 2. ORT `rc6_2b375_fp32` (`DFlash2 K=7`, `fp16_kv + FP32 GDN`) Before
vs. After
| Context (`Prompt + Gen`) | Baseline TTFT (`Prefill tok/s`) |
**Optimized TTFT (`Prefill tok/s`)** | Baseline Decode (`ms/cyc`) |
**Optimized Decode (`ms/cyc`)** |
|---|---:|---:|---:|---:|
| **`2K + 1K`** | `1.232 s` (`1,661.9`) | **`1.204 s` (`1,700.5
tok/s`)** | `62.78 tok/s` (`54.68 ms`) | **`81.20 tok/s` (`47.91 ms`,
`-12.4%`)** |
| **`8K + 1K`** | `4.607 s` (`1,778.1`) | **`4.352 s` (`1,882.3
tok/s`)** | `63.91 tok/s` (`54.44 ms`) | **`76.47 tok/s` (`48.65 ms`,
`-10.6%`)** |
| **`32K + 1K`** | `18.642 s` (`1,757.8`) | **`17.589 s` (`1,862.9
tok/s`)** | `58.34 tok/s` (`56.02 ms`) | **`65.65 tok/s` (`50.11 ms`,
`-10.5%`)** |
| **`128K + 1K`** | `81.111 s` (`1,616.0`) | **`77.106 s` (`1,699.9
tok/s`)** | `48.82 tok/s` (`58.37 ms`) | **`51.38 tok/s` (`53.52 ms`,
`-8.3%`)** |
| **`255K + 1K`** | `179.976 s` (`1,450.9`) | **`172.099 s` (`1,517.3
tok/s`)** | `51.17 tok/s` (`68.93 ms`) | **`50.52 tok/s` (`63.08 ms`,
`-8.5%`)** |
## Validation
- Multi-arch CUDA compilation verified cleanly across `sm_75`, `sm_80`,
`sm_86`, `sm_90`, and `sm_120f` with `-Werror all-warnings --Werror
default-stream-launch`.
- Speculative (`M = 8`) vs. `--force-reject` (`M = 1`) greedy
equivalence verified `128/128` (`100%` bit-identical) on both
`rc6_2b375_fp32` and `gguf_ud_iq3_xxs_q8_engram`.
- MMLU-Pro shard `0/80` (`10` items): `9/10` (`90%`, matching baseline
item-for-item) on `rc6_2b375_fp32` and `10/10` (`100%`) on
`gguf_ud_iq3_xxs_q8_engram` with `0` truncations.
- Added unit tests in `matmul_packed_precision_test.cc`,
`gguf_quant_packed_test.cc`, and `dynamic_sparse_attention_op_test.cc`.
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>