[CUDA] Reserve MoE GEMM profiler scratch outside the arena (#34167)
### Description
MoeGemmProfiler::runProfiling took its one-off profiling workspace from
the temp-space allocator with a stream-less Alloc(). With the default
CUDA EP that allocator is the BFC arena, which keeps every region it
grows. A stream-less request also cannot reuse a chunk that the compute
stream has used since (BFCArena::FindChunkPtr). So each MoE node that
profiles a new M bucket can leave another workspace-sized region in the
arena for the rest of the session.
This change reserves the scratch instead, the same way the fpA_intB GEMM
profiler already does (IAllocator::MakeUniquePtr with use_reserve = true
in contrib_ops/cuda/llm/gemm_profiler.h). BFCArena::Reserve takes the
memory straight from the device allocator and Free returns it.
Allocators without a Reserve override fall back to Alloc(), so they
behave as before. One example is the plain CUDAAllocator the CUDA plugin
EP uses for temp space.
Because reserved memory goes back to the device on Free, the workspace
guard now synchronizes the profiling stream before freeing. It is
declared after the stream guard, so a private stream outlives it on
every exit path, including exceptions from backend_.prepare() or a
failing tactic.
Test: MoeGemmProfilerTest.ProfilingScratchDoesNotGrowArena profiles
three nodes at two M buckets through a real BFC arena. On main the arena
grows by two extensions (3,145,728 bytes) with no reserves. With this
change it grows by 0 bytes and uses Reserve.
Measured on an A100 80GB, main (6596e0c865) vs this change:
- Multi-turn chat with Qwen3-30B-A3B-Instruct-2507 through ORT GenAI,
with prompts of 2,048, 8,192, 16,384 and 32,640 tokens in one session.
After turn 2 the model arena is 17,642 MiB instead of 66,282 MiB. Turn 4
completes instead of failing with out-of-memory, and the arena after
turn 4 is 23,786 MiB.
- Single 32,640-token prompt, builds run in A-B-B-A order, 10 warm
requests per build. Time to first token changes by +0.04% (95% CI -0.01%
to +0.09%) and decode by -0.04%. Generated tokens are identical, and
process peak is 47,266 MiB on main vs 46,946 MiB with this change.
- Profiling cost, on 48 same-shape QMoE nodes built from layer 0 of that
model: the first run, which includes all 96 profiling runs, takes 96 ms
longer (22,916 to 23,012 ms; 95% CI of the difference 64 to 129 ms; 10
interleaved pairs). Arena growth during that run drops from
1,610,612,736 to 536,870,912 bytes in every pair, and outputs are
identical.
### Motivation and Context
In a multi-turn chat with Qwen3-30B-A3B (48 QMoE layers) on an A100
80GB, each new prompt-length bucket made all 48 nodes profile. Each
node's roughly 770 MiB profiling request at the largest bucket left a 1
GiB arena region behind, so the arena grew by about 48 GiB in one turn
and the fourth turn ran out of memory.
Limits:
- Reserve() bypasses BFCArena binning and its gpu_mem_limit check. The
memory is still counted in the arena's total while profiling and
released right after, and the CUDA device memory budget, when
configured, still applies through CUDAAllocator.
- Each profiling run now does one cudaMalloc/cudaFree pair. Profiling
happens once per node and M bucket, and never during CUDA graph capture.
- Tested on SM80 Linux. CI covers SM86, Windows and the plugin EP build.
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>