onnxruntime

aeb1a16a - [CUDA] Add an opt-in FP8 DeepGEMM MoE decode path for QMoE (#32122)

[CUDA] Add an opt-in FP8 DeepGEMM MoE decode path for QMoE (#32122) ## Description Adds an opt-in DeepGEMM-based MoE decode path for QMoE, behind `ORT_QMOE_FP4_DEEPGEMM` (default **off**). The DSV4 MoE decode GEMMs are DRAM bound, so the prepacked weight format sets their cost; this path stores an fp8 e4m3 mirror of the MXFP4 weights with a power-of-two block scale and runs DeepGEMM's sm90 fp8 masked grouped kernel. Measured 11.5% lower decode step time on 8xH200 and 32 GiB per rank of weights freed relative to the bf16 mirror it replaces. ## Summary of Changes ### Build | File | Change | |------|--------| | `cmake/external/deep_gemm.cmake` | **New.** Header-only `FetchContent` of [deepseek-ai/DeepGEMM](https://github.com/deepseek-ai/DeepGEMM). | | `cmake/onnxruntime_providers_cuda.cmake` | Include the dependency and route `deep_gemm_sm90.cu` to the SM90 TMA target. | | `cmake/onnxruntime_providers_cuda_plugin.cmake` | Same wiring for the plugin EP build. | | `cmake/onnxruntime_cuda_source_filters.cmake` | Classify `deep_gemm_sm90.cu` as SM90-only. | ### Kernels and dispatch | File | Change | |------|--------| | `onnxruntime/contrib_ops/cuda/llm/moe_gemm/deep_gemm_sm90.{cu,h}` | **New.** Pack / masked grouped FC1 / interleaved SwiGLU / FC2 / unpack, with the validated 32-local-expert (eight-rank) configuration instantiated. | | `onnxruntime/contrib_ops/cuda/moe/qmoe_kernels.{cu,h}` | **New.** MXFP4 → fp8 e4m3 prepack with power-of-two block scales. | | `onnxruntime/contrib_ops/cuda/moe/moe_quantization.{cc,h}` | Gate and dispatch the path; the FP4 GEMV keeps priority when DeepGEMM is not selected. | | `onnxruntime/contrib_ops/cuda/llm/moe_gemm/moe_kernels.{cu,h}` | Supporting entry points. | | `docs/contrib_ops/cuda/moe_qmoe.md` | Document the env var, the weight format, and the memory cost. | ## Testing - `python test_qmoe_fp4_cuda.py` — passes with the path both enabled and disabled. - **Weight conversion is bit exact.** An E2M1 code carries at most two significant bits and e4m3 carries four, so a *power-of-two* block scale only shifts exponents and never disturbs a mantissa. Checked over all 256 experts of all 46 layers: every weight reproduces the fp32 dequantization bitwise, with no underflow and no clipping. - **Padded-row skip verified against a poisoned workspace** (NaN, ±Inf, 1e30 across 128 patterns, ragged and zero counts): the compact output is identical even though 40–60% of padded FC2 rows become NaN. - MMLU-Pro (800 samples) 0.660 vs 0.6625 baseline — 2 disagreements, McNemar p=0.48. Rank argmax disagreements 0. - Builds clean with and without NCCL; `deep_gemm_sm90.cu` compiles into the SM90 TMA target. ## Motivation and Context Per decode step on 8xH200, prompt 1024 / generation 512: | Metric | Before | After | |---|---|---| | Step time | 27.59 ms | 24.43 ms (−11.5%) | | FC1 GEMM (standalone) | 70.9 us | 37.9 us | | FC2 GEMM (standalone) | 38.6 us | 22.3 us | Smaller tiles also soften the wave-quantization cliff: going from 8 to 9 active experts costs bf16 35% but fp8 only 18%. Two things reviewers should know: 1. **The conventional `amax/448` block scale is not usable here.** It is not a power of two, so every weight would acquire a full mantissa before being rounded back to three bits (4.8% max relative error). The power-of-two scale is load-bearing for exactness, not an optimization. Activations do use `amax/448` per token per 128-channel chunk, where the full e4m3 range is worth more than an exact scale. 2. **The path is off by default and gated to a narrow shape.** It requires sm90, SwiGLU fusion, no bias, and matching hidden/inter sizes. At four ranks (64 experts), the e4m3 mirror would cost 64.5 GiB per rank over 43 layers and exhaust memory during PrePack, so this configuration is excluded from the DeepGEMM gate and continues through standard QMoE. This is documented in `moe_qmoe.md`. The main reviewer-facing consideration is the new third-party `FetchContent` dependency; it is header-only and confined to the SM90 build. ## Checklist - [x] Tests added/updated - [x] No breaking changes (new path is opt-in and off by default) - [x] Documentation updated