onnxruntime

bceab03c - [CUDA] Default QMoE GEMV fp16 accumulation for fp16 activations (#29166)

[CUDA] Default QMoE GEMV fp16 accumulation for fp16 activations (#29166) ### Description Make fp16 accumulation the default for the CUDA QMoE GEMV fast path when activations are fp16. The previous fp32 accumulation behavior remains available as an opt-in fallback with `ORT_MOE_GEMV_FP32_ACCUM=1`, and bf16 activations continue to use fp32 accumulation. This is motivated by GPT-OSS-20B decode measurements where fp16 accumulation was close in accuracy to the fp32 path and materially faster. ### Changes - Invert the QMoE GEMV accumulation environment knob: - default fp16 accumulation for fp16 activations - `ORT_MOE_GEMV_FP32_ACCUM=1` restores fp32 accumulation - bf16 stays on fp32 accumulation - Document the new runtime knob in the QMoE CUDA docs. - Add the standalone helper, full-model decode, and MMLU smoke measurements to the QMoE GEMV experiment log. ### Measurements | Measurement | Default fp16 accumulation | `ORT_MOE_GEMV_FP32_ACCUM=1` | |---|---:|---:| | Standalone GPT-OSS QMoE helper latency | 0.0708 ms | 0.0812 ms | | Helper FC1 SwiGLU GEMV avg | 13.93 us | 21.57 us | | Helper FC2 GEMV avg | 10.14 us | 12.24 us | | Full GPT-OSS CUDA-graph decode latency | 2.588930 ms/token | 2.827260 ms/token | | Full GPT-OSS CUDA-graph decode throughput | 386.259956 tok/s | 353.699315 tok/s | The full-model A/B shows about +9.2% decode throughput for the default fp16 accumulation path versus the fp32 fallback in this run. ### Accuracy Prior 1000-sample MMLU smoke runs matched pooled accuracy for both modes: | Mode | Pooled accuracy | |---|---:| | fp32 accumulation | 0.8260 | | fp16 accumulation | 0.8260 | ### Testing - `lintrunner -a onnxruntime/contrib_ops/cuda/llm/moe_gemm/moe_gemv.cu` - `cmake --build /home/tianlei/onnxruntime/build/cu130/Release --target onnxruntime_providers_cuda --parallel $(nproc)` - `git diff --check -- onnxruntime/contrib_ops/cuda/llm/moe_gemm/moe_gemv.cu docs/contrib_ops/cuda/qmoe_gemv_experiments.md docs/contrib_ops/cuda/moe_qmoe.md` - Standalone QMoE helper A/B on `gpt_oss_20b_m1_top4_fp16_2880x2880_e32` - Full GPT-OSS CUDA-graph decode A/B