onnxruntime
f0843448 - [CUDA] Avoid repacking NVFP4 QMoE decode weights (#32325)

Commit
8 days ago
[CUDA] Avoid repacking NVFP4 QMoE decode weights (#32325) ## Description Make the NVFP4 QMoE decode GEMV consume the schema-native packed weights and scales directly. This removes the decode-only weight repack and activation-dtype scale bank while retaining the existing grouped-GEMM paths for larger workloads. ## Summary of Changes ### Raw NVFP4 GEMV | File | Change | |------|--------| | `onnxruntime/contrib_ops/cuda/llm/moe_gemm/moe_gemv_fp4.cu` | Add a raw N-packed NVFP4 GEMV that decodes E2M1 weights and E4M3 scales in-register, including fused SwiGLU. | | `onnxruntime/contrib_ops/cuda/llm/moe_gemm/moe_gemv_fp4.h` | Extend support checks and launch interfaces for the raw NVFP4 layout. | | `onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc` | Retain and dispatch directly from raw NVFP4 initializers instead of building decode-specific packed weights and combined scales. | | `onnxruntime/contrib_ops/cuda/llm/moe_gemm/moe_kernels.cu` | Extend the fused expert-map prologue to Qwen's `top_k=10` configuration. | ### Tests - Add the Qwen Flash decode shape and verify that it selects the GEMV route. - Cover disabled prepacking and fallback behavior above the GEMV row limit. ## Testing - `python3 -m pytest -q onnxruntime/test/python/transformers/test_qmoe_nvfp4_cuda.py` - 32 passed on NVIDIA H200. - CUDA provider target built successfully with FP4 QMoE enabled. - `ruff check onnxruntime/test/python/transformers/test_qmoe_nvfp4_cuda.py` ## Checklist - [x] Tests added/updated - [x] No breaking changes - [x] Documentation not required
Author
Parents
Loading