onnxruntime
1f539821 - [CUDA] Reserve MoE GEMM profiler scratch outside the arena (#34167)

Commit
1 day ago
[CUDA] Reserve MoE GEMM profiler scratch outside the arena (#34167) ### Description MoeGemmProfiler::runProfiling took its one-off profiling workspace from the temp-space allocator with a stream-less Alloc(). With the default CUDA EP that allocator is the BFC arena, which keeps every region it grows. A stream-less request also cannot reuse a chunk that the compute stream has used since (BFCArena::FindChunkPtr). So each MoE node that profiles a new M bucket can leave another workspace-sized region in the arena for the rest of the session. This change reserves the scratch instead, the same way the fpA_intB GEMM profiler already does (IAllocator::MakeUniquePtr with use_reserve = true in contrib_ops/cuda/llm/gemm_profiler.h). BFCArena::Reserve takes the memory straight from the device allocator and Free returns it. Allocators without a Reserve override fall back to Alloc(), so they behave as before. One example is the plain CUDAAllocator the CUDA plugin EP uses for temp space. Because reserved memory goes back to the device on Free, the workspace guard now synchronizes the profiling stream before freeing. It is declared after the stream guard, so a private stream outlives it on every exit path, including exceptions from backend_.prepare() or a failing tactic. Test: MoeGemmProfilerTest.ProfilingScratchDoesNotGrowArena profiles three nodes at two M buckets through a real BFC arena. On main the arena grows by two extensions (3,145,728 bytes) with no reserves. With this change it grows by 0 bytes and uses Reserve. Measured on an A100 80GB, main (6596e0c865) vs this change: - Multi-turn chat with Qwen3-30B-A3B-Instruct-2507 through ORT GenAI, with prompts of 2,048, 8,192, 16,384 and 32,640 tokens in one session. After turn 2 the model arena is 17,642 MiB instead of 66,282 MiB. Turn 4 completes instead of failing with out-of-memory, and the arena after turn 4 is 23,786 MiB. - Single 32,640-token prompt, builds run in A-B-B-A order, 10 warm requests per build. Time to first token changes by +0.04% (95% CI -0.01% to +0.09%) and decode by -0.04%. Generated tokens are identical, and process peak is 47,266 MiB on main vs 46,946 MiB with this change. - Profiling cost, on 48 same-shape QMoE nodes built from layer 0 of that model: the first run, which includes all 96 profiling runs, takes 96 ms longer (22,916 to 23,012 ms; 95% CI of the difference 64 to 129 ms; 10 interleaved pairs). Arena growth during that run drops from 1,610,612,736 to 536,870,912 bytes in every pair, and outputs are identical. ### Motivation and Context In a multi-turn chat with Qwen3-30B-A3B (48 QMoE layers) on an A100 80GB, each new prompt-length bucket made all 48 nodes profile. Each node's roughly 770 MiB profiling request at the largest bucket left a 1 GiB arena region behind, so the arena grew by about 48 GiB in one turn and the fourth turn ran out of memory. Limits: - Reserve() bypasses BFCArena binning and its gpu_mem_limit check. The memory is still counted in the arena's total while profiling and released right after, and the CUDA device memory budget, when configured, still applies through CUDAAllocator. - Each profiling run now does one cudaMalloc/cudaFree pair. Profiling happens once per node and M bucket, and never during CUDA graph capture. - Tested on SM80 Linux. CI covers SM86, Windows and the plugin EP build. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Author
Parents
Loading