onnxruntime
54cbb581 - [CUDA] Size QMoE split-K SwiGLU GEMV partials by GEMV rows, not prefill rows (#34168)

Commit
2 days ago
[CUDA] Size QMoE split-K SwiGLU GEMV partials by GEMV rows, not prefill rows (#34168) ### Description CutlassMoeFCRunner reserves an FP32 split-K partials buffer for the fused SwiGLU MoE GEMV (FP16 activations, INT4 weights, swiglu_fusion == 1, see MoeGemvSplitK2SwiGLUSupported) and sized it for every expanded row of the call. Its only reader is the split-K two-pass GEMV. That path runs only when is_moe_gemv_supported() accepts the problem, which needs expanded_num_rows <= moe_gemv::kMaxProfiledExpandedRows (80), and only with FP32 GEMV accumulation (ORT_MOE_GEMV_FP32_ACCUM=1). Above 80 expanded rows the grouped GEMM runs and the buffer is never read, so at prefill sizes the workspace carried a buffer that grows with the prompt and is never used. This change caps the partials at 80 expanded rows. Calls with up to 80 expanded rows get exactly the same size as before; larger calls keep a fixed 2 * 80 * 2 * inter_size * 4 bytes (0.94 MiB for inter_size 768). Tests: - QMoEWorkspaceTest.SplitKSwiGluPartialsDoNotScaleWithPrefillRows fails on main (per-token growth at GEMV sizes 176,640 B, needs at least 225,857 B) and passes with this change. - QMoEWorkspaceTest.SplitKSwiGluPartialsCoverGemvRows checks that GEMV-sized calls still get the full buffer. Validation on A100 80GB (SM80), using layer 0 of Qwen3-30B-A3B-Instruct-2507 (INT4, block 128, top-8, 128 experts) as a standalone QMoE model, main vs this branch: - Outputs are bitwise identical at 8, 80, 88 and 1024 expanded rows, both with ORT_MOE_GEMV_FP32_ACCUM=1 and with the default. - With ORT_MOE_GEMV_FP32_ACCUM=1, Nsight Systems shows moe_gemv_splitk_partials_kernel and moe_gemv_splitk_reduce_swiglu_kernel launching at 8 and 80 expanded rows, and no GEMV kernels at 88 and 1024. - compute-sanitizer memcheck reports 0 errors on those runs. memcheck alone would not catch an undersized buffer, because the partials share one workspace allocation with the other buffers. As a control, halving the cap changes the output at 80 expanded rows while memcheck still reports 0 errors, so the parity check is the meaningful one. - End to end with the full model through ORT GenAI at a 32,640-token prompt and ORT_MOE_GEMV_FP32_ACCUM=1 (one process per build): process peak 47,248 MiB on main vs 43,152 MiB with this change, identical generated tokens, decode 126.2-126.6 vs 126.3-126.6 tokens/s. The buffer is reserved the same way with the default accumulation. An earlier run of the same change on main at e4234e5617 measured a peak of 47,254 vs 43,158 MiB in that mode (two processes per build, identical in each). ### Motivation and Context For Qwen3-30B-A3B at a 32,640-token prompt the partials buffer was 2 * 261,120 * 1,536 * 4 B = 3,059 MiB of workspace that was never read. Limits: - Only the FP16 + INT4 + fused-SwiGLU runner allocates this buffer, so other dtypes and activations are unaffected. - The small capped buffer is still reserved when FP32 accumulation is off. Dropping it entirely would tie the workspace layout to that environment variable, so this PR leaves it. - Tested on SM80 Linux only. CI covers SM86 and Windows. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Author
Parents
Loading