onnxruntime
6aa9b274 - [CUDA] Extend MoEPacked for top-k, dispatch and fused MoE calculation (#33963)

Commit
4 days ago
[CUDA] Extend MoEPacked for top-k, dispatch and fused MoE calculation (#33963) ### Description <!-- Describe your changes. --> Extends CUDA `com.microsoft::MoEPacked` with ReLU-square experts and full-score routing. Routing scores and mixing weights can be FP32 independently of the FP16/BF16 expert activations. This PR does not change QMoE. When `router_weights` is provided, `router_probs` contains full selection scores and `router_weights` contains separate mixing values. `MoEPacked` selects the top-k experts from the scores, gathers their mixing values, optionally normalizes them, and combines the expert outputs. Without `router_weights`, the existing softmax-logit routing path is used. ```text Hidden states [N, H] ───────────────────────────┐ v Separate router → selection scores [N, E] → MoEPacked → output [N, H] └→ mixing weights [N, E] ────────┘ ``` Routing tensors share a dtype: FP32 or the activation dtype. Full-score routing supports up to 32 experts per token; the legacy softmax path supports up to 16. Packed expert-weight layouts are unchanged. The `relu2` expert activation (`relu_squared` is an alias) requires IQ2_XXS gate and down weights, with up weights omitted. Learned router projection remains outside `MoEPacked`; this operator has no integrated router projection or `router_scale` attribute. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> This allows a separate router to provide full-precision selection scores and mixing values to `MoEPacked` without first narrowing them to the expert activation dtype. It also enables the supported IQ2_XXS ReLU-square expert path while keeping router computation and packed expert execution separate. ### Normalization-default compatibility correction Commit `18da59fe12` restores `normalize_routing_weights=1` in the schema and CUDA defaults, preserving legacy models that omit the attribute. Full-score callers wanting unnormalized mixing values must explicitly set `0`. The full-score tests now do so; a new omitted-attribute, `k=1` legacy regression protects the default. The generated operator documentation records default `1`. - Owning layer: all 18 production routing tests plus one fresh-schema/kernel execution guard passed (19 checks, zero failures or skips). - Integrated #34003 stack: all 22 production routing tests plus the same guard passed (23 checks, zero failures or skips). - The baseline-schema negative control failed only the new legacy regression: expected `0.28125`, actual `0.140625`. The explicit full-score opt-out still passed. - Profiles recorded 71 owning-layer and 190 integrated `CUDAExecutionProvider/MoEPacked` executions. Changed-range clang-format 20.1.8, whitespace, and scoped official documentation-rendering checks passed. Both positive runs freshly rebuilt the operator, schema and tests against ABI-matched retained ORT support. This is not a clean full-source build, full CI approval, full hybrid-offload qualification, or full-model acceptance. Full `--gen_doc validate` was not run. ### Historical validation before the normalization-default correction - All 17 production `MoEPackedRoutingTest` cases and three additional scope guards passed without a test filter: 20 tests, zero failures or skips. - 70 profiled `MoEPacked` executions ran on CUDA. Head `87daac143f` removes the obsolete QMoE helper and four QMoE tests, preserving every MoEPacked test and leaving runtime code unchanged. The branch remains rebased onto main `0b22d27a85`, preserving the hybrid-offload implementation from #33973. The new activation/routing modes remain GPU-resident; unsupported CPU-weight-bank combinations are rejected explicitly rather than entering the legacy hybrid-offload route. Qualification freshly rebuilt the current schema/test executable against the unchanged, previously qualified MoEPacked module and retained native support libraries. This is not a clean full-source runtime build, full hybrid-offload qualification, full CI approval, or independent full-model parity. Full `--gen_doc validate` was not run. --------- Signed-off-by: rui-ren <ruiren1225@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f4d0e1a3-a389-4cc8-9c85-0fd0e3ffce5b Copilot-Session: c8741c5a-3e9e-4860-b846-7d6fb0025040
Author
Parents
Loading