onnxruntime

d9c279b7 - P0/P1: packed-code foundation and INT3 QMoE v1 contract (#33394)

P0/P1: packed-code foundation and INT3 QMoE v1 contract (#33394) ## Summary Implements the P0 reference/tooling and P1 QMoE v1 contract/host-validation slices of the [INT3 QMoE delivery plan](https://github.com/microsoft/onnxruntime/blob/2b1185e24a5a79c56fd01226ad81b016450d0ef3/docs/design/int3-qmoe-end-to-end-delivery-plan.md). Related roadmap: #32657. No P2 CUDA kernels are included. ### P0: Portable Format and Numerical References - Generic row-local LSB-first UINT8 pack/unpack for widths 1..8, checked size arithmetic, and separate canonical-padding validation. INT5/INT6 utility coverage does not enable those widths in QMoE. - INT3 offset-4 decoding, deterministic fixture RTN with stored-scale rounding and subnormal-underflow reporting, and handwritten byte/numerical vectors covering row/expert boundaries, packed block zero points, and a tiny predetermined-route SwiGLU MoE. - Independent scalar oracles and standard opset-18 ONNX unpack/decode graphs; no draft Pack/Unpack operator is required. - Self-contained reference tests use local golden vectors and installed standard ONNX operators; no external ONNX source checkout or pinned ONNX commit is required. - Reference pytest coverage runs in the existing Python quantization CI stage; CMake copies the helper modules and golden JSON fixture into the build output. No feature-specific workflow is added. ### P1: Compatible QMoE v1 Extension - INT3 is added to QMoE version 1. There is no QMoE v2 registration and no Microsoft-domain available/released-version increase. - Models without an effective INT3 projection keep legacy shape/type propagation. Existing 2/4/8-bit byte semantics and attribute defaults are preserved, including expert_weight_bits=4, quant_type="int", and weights_prepacked=-1. - INT3 requires integer quantization, explicit weights_prepacked=0, activation_type="swiglu", fused interleaved gate/up, block sizes 32/64/128, block-aligned H/I, and matching FP16/BF16 activation/scale types. Omitting quant_type uses the existing "int" default; exporters should still emit it explicitly. - Support independent projection metadata, including (3,4), (3,3), and (4,3), and packed per-row block zero points. Infer H from activation shape and I from fused FC1 output rows, not FC2 packed bytes. Check shapes, types, symbolic dimensions, byte counts, and overflow. - Shared host-only ValidateInt3ProjectionContent requires block-aligned K (not N) at the projection level and checks exact buffer sizes, finite nonnegative FP16/BF16 scales, all-4 explicit zero points, canonical padding, and checked dimensions. Padded byte-count arithmetic is overflow-safe without changing the strict byte-aligned helper. - Native regressions verify that legacy INT4 models resolve to QMoE v1 and find the existing CPU kernel both with and without an explicit Microsoft-domain import. Adjacent Microsoft operators remain at v1. - Generated QMoE documentation is synchronized; non-QMoE documentation is unchanged. ## Validation of the v1 Revision Commit eda735324e was validated inside Docker jiafa-dev: - Offline Python reference tests: **223 passed, 1 skipped**. The historical optional external-checkout case was removed in 6c875aef57; current results are listed below. ONNX 1.23.2, NumPy 2.4.6, ml_dtypes 0.6.0; BF16 coverage executed. - Actual current ORT schema, registration entry point and QMoE test sources compiled using cached Ninja-generated flags and linked with cached ORT libraries: **17 passed** (11 schema/registry tests and 6 content/byte-helper tests). - C++ filter: QMoEShapeInferenceTest.*:QMoEFormatValidationTest.*. New test files are collected by the existing contrib-test CMake glob. - Pinned clang-format 20.1.8, Ruff 0.12.12 check/format, editor diagnostics and git whitespace checks passed. - QMoE description was checked byte-for-byte using the documentation generator's rendering rules; all other operator documentation is byte-identical to the preceding PR head. - This is isolated current-source compilation/linking, **not a clean full ORT build** or CUDA inference qualification. The cached build predates two neighboring schemas, so their current definitions were also compiled for linking. Container CUDA runtime paths and system -lrt replaced unavailable cached paths. Only the existing MLAS GNU-stack linker warning remained. - Hosted CI for this new head is pending. Previous external-reference interoperability results are historical; that opt-in test was not rerun for this schema-only revision. ## Scope and Review Gates This PR adds **no INT3 execution kernel**. Existing v1 CPU/CUDA/WebGPU registrations remain; schema acceptance does not establish INT3 inference support. Provider validation/execution, packed CUDA decode, bounded prefill, export and model qualification belong to subsequent work. The end-to-end follow-up now builds on this v1 revision; see the coordinated Draft PRs below. The host content validator has **no runtime call sites yet**: it accepts caller-supplied host buffers and is directly unit-tested. Schema inference checks metadata, not weight contents; model loading does not automatically scan scales or zero points. Python-only tails/empty dimensions/FP32-scale fixtures do not broaden the block-aligned FP16/BF16 operator profile. The v1-compatible extension approach supersedes this PR's earlier v2/domain-version proposal. Operator/exporter owner review and format freeze remain required. No approved ONNX datatype, standard operator, production quantizer/exporter, checkpoint importer, accuracy or performance claim is made. ## Scoped Checkpoint Audit The local Olive audit at a9290ab9bf668cae5a02ad7473c8cb0f5476d35e found WeightQuantizer and pack_to_uint8 restricted to 2/4/8 bits, with packing_factor = 8 // bits; the inspected GPTQ pass imports that quantizer, and PrecisionBits has no three-bit option. These inspected paths do not establish a compatible native INT3 checkpoint source. External GPTQ implementations and GGUF Q3_K/IQ3 were not audited. Future importers must verify code mapping, bit order, tensor orientation, grouping, scales, zero points, permutations, gate/up interleaving and FC2 orientation rather than relabel unknown checkpoint bytes. Unrelated local files and submodule changes are excluded. ## Follow-up End-to-End Work The experimental end-to-end implementation now builds on this PR's compatible QMoE v1 contract. The work is split across three coordinated **Draft PRs**: - **ORT runtime: [microsoft/onnxruntime#33519](https://github.com/microsoft/onnxruntime/pull/33519)**, stacked directly on this PR. Adds CUDA execution for `(FC1, FC2)=(3,3), (3,4), (4,3)` with FP16/BF16, direct packed extraction in both projections, routing and fused SwiGLU, strict validation, bounded activation tiling, tests, and experimental model results. - **Olive quantization: [microsoft/Olive#2749](https://github.com/microsoft/Olive/pull/2749)**. Adds symmetric grouped INT3 RTN with offset-4 codes, stored-scale rounding, canonical row-local three-bit packing/unpacking, and ceil-bit storage shapes. - **Mobius export: [onnxruntime/mobius#783](https://github.com/onnxruntime/mobius/pull/783)**. Enables expert-only INT3 overrides for the three width pairs while retaining the model-wide INT4 fallback, using existing native fused QMoE graph and external-data plumbing. Together these changes exercised source checkpoint -> Olive quantization -> Mobius ONNX export -> ORT CUDA inference. Six small cross-project cases cover both activation dtypes and all three width pairs. Full Qwen3-30B-A3B exports and model measurements cover all routed expert gate/up/down weights at INT3 `(3,3)` versus matched INT4 `(4,4)`, with 48 QMoE layers, block64, FP16, attention/embedding INT4 and head INT8. Graph-plus-weight size is 12,782,047,512 versus 16,405,926,354 bytes, a **22.09% reduction**. The runtime PR records A100 measurements at input lengths 128/512/2048/4096 and output length 32. **Current INT3 inference is substantially slower than INT4; this is not a performance delivery.** The paired 5-shot MMLU pipeline check uses only one question per subject (57 total): INT3 40/57 and INT4 49/57, with matching document/prompt/target hashes. These are **not full MMLU scores**; no completed full-MMLU result is claimed here. ### Delivery Plan Status These follow-ups do **not** complete planned PR2 through PR4: | Planned stage | Implemented evidence | Remaining gate | | --- | --- | --- | | PR2: CUDA correctness | Independent references, bounded execution, actual CUDA Session coverage | Clean provider-target build and formal integration-test acceptance | | PR3: packed decode | Direct packed FC1/FC2 execution for all three pairs | Versioned provider-private prepack, cache identity/lifetime and prepack-cost qualification | | PR4a: baseline prefill | Functional bounded direct-packed prefill, including full-model long prompts | Required selected-expert/tiled dequantization plus existing GEMM and dedicated acceptance tests | | PR4b/4c: optimized prefill | Not delivered | Winning kernel candidate, integration and performance acceptance | Before submission, the follow-up runtime's 82 actual CUDA Session cases passed; Olive's 176 scoped quantization tests and Mobius's 18 new tests also passed. Runtime experiments reused isolated cached build artifacts with explicit ABI alignment, **not a clean full build**. Formal OpTester execution, capture, concurrency, lifetime and full-model qualification remain open. Full-model mixed-pair qualification is also open; small fixtures do not close that gate. See #33519 and its committed results report/JSON for protocols, provenance, measurements and limitations. The checkpoint audit above describes the inspected pre-change Olive revision; #2749 supplies the subsequent producer changes. None of these follow-ups changes this PR's P0/P1-only diff or constitutes ONNX standard approval or production qualification. ## Review Fixes and Padding Experiment Follow-up Commit e45296be3b integrates the no-exceptions assertion guards, symbolic projection-row consistency fix and regression cases, and unpacked zero-point code clarification directly into this PR. Rebuilt current-source normal schema/format tests: **17/17 passed**; format tests compiled with ORT_NO_EXCEPTIONS plus the real helper implementation: **6/6 passed**. Pinned clang-format, whitespace and schema/doc synchronization checks passed. These remain isolated cached-dependency builds, not clean full or globally exception-disabled builds. Padding experiments remain separately reviewable in #33530: two corrected full A100/CUDA13 runs, 108 cases each, with all128expert rotation covered per full trial. For block64,24->32byte padding adds33.33% weight payload and is approximately11% slower for decode projection and10% slower for32token projection in the tested scalar-load implementation. This is not a whole-model or pure-bandwidth conclusion and does not rule out future vectorized private layouts. The padding design thread remains open for reviewer acceptance; experiment source and raw records are not added to this P0/P1 diff. ## Existing CI Integration Commit 8a41267fc0 removes the dedicated packed-reference workflow and runs the suite from the existing Python quantization test stage in tools/ci_build/build.py, after unittest discovery when contrib ops are enabled. CMake copies the complete reference directory and golden_vectors.json into the test build output. Explicit pytest execution is necessary because unittest discovery does not collect these module-level pytest functions. Existing Python CI already installs pytest; no new workflow or dependency-install job is introduced. Validated by copying the directory with cmake -E copy_directory and executing the exact new pytest command extracted from build.py: **223 passed, 1 skipped** at 8a41267fc0 (historical optional checkout test, removed in 6c875aef57). Ruff check/format and git whitespace checks passed. Hosted CI for this head is not yet verified. ## Latest Review Fixes Commit 6c875aef57 addresses the latest review comments: - Removes the pinned external ONNX reference checkout test entirely. The suite is self-contained: **223 passed**, with no external-checkout skip. The ONNX submodule gitlink is unchanged by this PR. - Requires explicit `activation_type="swiglu"` for the INT3 fused profile and rejects the default/non-SwiGLU cases. Legacy INT4 activation behavior remains covered. Schema documentation is synchronized. - Removes the projection-level N-alignment restriction (K remains block-aligned), removes unreachable weight-tail validation, and computes each expected zero-point byte once before checking rows. Adds an N=127 acceptance regression. - Adds standard copyright/license headers and trailing newlines, restores the unrelated CPU model test to the main-branch version, and fixes CMake COMMAND indentation. Recompiled current affected C++ sources with cached dependency libraries: **18/18 normal schema/format tests passed** and **7/7 format tests passed with ORT_NO_EXCEPTIONS and the real helper implementation**. These are scoped cached-dependency builds, not a clean whole-ORT or globally no-exceptions build. The exact existing-CI pytest command also passed after CMake-copying the suite into a temporary build-output directory. Pinned clang-format, Ruff check/format, schema/doc synchronization and git whitespace checks passed. Hosted CI for this new head has not yet been verified. --------- Signed-off-by: David Fan <jiafa@microsoft.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>