onnxruntime

db4a8b1e - [CUDA] Document INT2 numerical limits and harden QMoE tests (#33858)

[CUDA] Document INT2 numerical limits and harden QMoE tests (#33858) ## Description Follow up on the three nonblocking comments from @baijumeswani in #33787. Document the intentional extreme-value limits of INT2 CPU and BF16 Tensor-Core arithmetic, and strengthen the regression coverage without changing the existing numerical policy. ## Summary of Changes | Review feedback | Change | |---|---| | [BF16 lower-range limit](https://github.com/microsoft/onnxruntime/pull/33787#discussion_r4462264678) | Document that halving a centered `+1` weight at scale `2^-133` rounds it to zero before MMA; add an explicit GPU regression distinguishing this result from the nonzero SIMT reference. Do not claim full-range BF16 support. | | [CPU unscaled-product overflow](https://github.com/microsoft/onnxruntime/pull/33787#discussion_r4462264682) | Document the scale-after-accumulation limit and add scalar/runtime-dispatch regressions for a finite neighboring case and the reported overflow case with mathematical result `-8`. | | [Nonzero reference, actual dispatch, positive threshold override](https://github.com/microsoft/onnxruntime/pull/33787#discussion_r4462264685) | Calculate an independent nonzero SwiGLU reference, assert successful INT2 Tensor-Core launches for both projections, and cover positive row-threshold overrides at 24 routes and a threshold above the 40-route workload. | - Gate successful-launch diagnostics behind the existing `ORT_QMOE_SHARED_RESIDENT_LOG` flag, cached on first use; the default execution path does not format or emit diagnostic strings. - Exercise standard GPU-only, fully resident offload, and hybrid CPU/GPU configurations. Keep the existing promotion-alignment and FC3-bit checks in a separate test. - Preserve the current CPU and CUDA arithmetic; these changes characterize the documented limits rather than introducing a new extreme-value fallback. ## Testing Local validation on Linux with CUDA 13.0 and NVIDIA A100: - Rebuilt `onnxruntime_providers_cuda` and `onnxruntime_provider_test`. - `QMoESharedLayoutTest.*`: **30 passed, 4 skipped**; AVX-512 tests were skipped because the host does not support AVX-512. The new scalar/AVX2 overflow regression passed. - Three Python tests passed: - `TestQMoEOffloadV4.test_v4_int2_bf16_tensorcore_scale_limits_and_dispatch` (16 numerical/dispatch scenarios). - `TestQMoEOffloadV4.test_v4_int2_bounce_alignment_and_fc3_bits`. - `TestQMoEOffloadV4.test_v4_int4_and_int3_offload_modes_and_numerical_parity`. - Negative controls confirmed that the strengthened regression rejects both mocked all-zero outputs and a numerically correct forced SIMT fallback. - Changed-source `lintrunner` and `git diff --check` passed. To reproduce, use the rebuilt Python package rather than an older installed wheel: ```bash cd build/cu130/Release ./onnxruntime_provider_test --gtest_filter='QMoESharedLayoutTest.*' PYTHONPATH="$PWD" ../../../.venv/bin/python \ ../../../onnxruntime/test/python/test_qmoe_offload_v4.py \ TestQMoEOffloadV4.test_v4_int2_bf16_tensorcore_scale_limits_and_dispatch \ TestQMoEOffloadV4.test_v4_int2_bounce_alignment_and_fc3_bits \ TestQMoEOffloadV4.test_v4_int4_and_int3_offload_modes_and_numerical_parity -v ``` Windows, CUDA EP plugin execution, AVX-512 execution, and H200 full-model benchmarks were not run locally; platform CI remains required. ## Checklist - [x] Tests added/updated - [x] No breaking changes - [x] Documentation updated Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>