[CUDA] Extend MoEPacked for top-k, dispatch and fused MoE calculation (#33963)
### Description
<!-- Describe your changes. -->
Extends CUDA `com.microsoft::MoEPacked` with ReLU-square experts and
full-score routing. Routing scores and mixing weights can be FP32
independently of the FP16/BF16 expert activations. This PR does not
change QMoE.
When `router_weights` is provided, `router_probs` contains full
selection scores and `router_weights` contains separate mixing values.
`MoEPacked` selects the top-k experts from the scores, gathers their
mixing values, optionally normalizes them, and combines the expert
outputs. Without `router_weights`, the existing softmax-logit routing
path is used.
```text
Hidden states [N, H] ───────────────────────────┐
v
Separate router → selection scores [N, E] → MoEPacked → output [N, H]
└→ mixing weights [N, E] ────────┘
```
Routing tensors share a dtype: FP32 or the activation dtype. Full-score
routing supports up to 32 experts per token; the legacy softmax path
supports up to 16. Packed expert-weight layouts are unchanged.
The `relu2` expert activation (`relu_squared` is an alias) requires
IQ2_XXS gate and down weights, with up weights omitted. Learned router
projection remains outside `MoEPacked`; this operator has no integrated
router projection or `router_scale` attribute.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
This allows a separate router to provide full-precision selection scores
and mixing values to `MoEPacked` without first narrowing them to the
expert activation dtype. It also enables the supported IQ2_XXS
ReLU-square expert path while keeping router computation and packed
expert execution separate.
### Normalization-default compatibility correction
Commit `18da59fe12` restores `normalize_routing_weights=1` in the schema
and CUDA defaults, preserving legacy models that omit the attribute.
Full-score callers wanting unnormalized mixing values must explicitly
set `0`. The full-score tests now do so; a new omitted-attribute, `k=1`
legacy regression protects the default. The generated operator
documentation records default `1`.
- Owning layer: all 18 production routing tests plus one
fresh-schema/kernel execution guard passed (19 checks, zero failures or
skips).
- Integrated #34003 stack: all 22 production routing tests plus the same
guard passed (23 checks, zero failures or skips).
- The baseline-schema negative control failed only the new legacy
regression: expected `0.28125`, actual `0.140625`. The explicit
full-score opt-out still passed.
- Profiles recorded 71 owning-layer and 190 integrated
`CUDAExecutionProvider/MoEPacked` executions. Changed-range clang-format
20.1.8, whitespace, and scoped official documentation-rendering checks
passed.
Both positive runs freshly rebuilt the operator, schema and tests
against ABI-matched retained ORT support. This is not a clean
full-source build, full CI approval, full hybrid-offload qualification,
or full-model acceptance. Full `--gen_doc validate` was not run.
### Historical validation before the normalization-default correction
- All 17 production `MoEPackedRoutingTest` cases and three additional
scope guards passed without a test filter: 20 tests, zero failures or
skips.
- 70 profiled `MoEPacked` executions ran on CUDA.
Head `87daac143f` removes the obsolete QMoE helper and four QMoE tests,
preserving every MoEPacked test and leaving runtime code unchanged.
The branch remains rebased onto main `0b22d27a85`, preserving the
hybrid-offload implementation from #33973. The new activation/routing
modes remain GPU-resident; unsupported CPU-weight-bank combinations are
rejected explicitly rather than entering the legacy hybrid-offload
route.
Qualification freshly rebuilt the current schema/test executable against
the unchanged, previously qualified MoEPacked module and retained native
support libraries. This is not a clean full-source runtime build, full
hybrid-offload qualification, full CI approval, or independent
full-model parity. Full `--gen_doc validate` was not run.
---------
Signed-off-by: rui-ren <ruiren1225@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f4d0e1a3-a389-4cc8-9c85-0fd0e3ffce5b
Copilot-Session: c8741c5a-3e9e-4860-b846-7d6fb0025040