onnxruntime
c5300b71 - Bound QMoE workspace with configurable row tiling (#32097)

Commit
1 day ago
Bound QMoE workspace with configurable row tiling (#32097) ## Summary Add optional row tiling to the CUDA grouped-QMoE path to bound temporary workspace usage for larger inputs. Tiling is disabled by default, preserving the existing execution path. Applications can configure the maximum rows per tile for each ORT session: ```json "session_options": { "ep.cuda.qmoe_row_tile_size": "64" } ``` Each tile runs sequentially on the same CUDA stream and reuses its routing metadata and runner workspace. ## Changes - Add row-tile planning and bounded scratch-layout helpers. - Reuse the grouped-MoE workspace across sequential row tiles. - Add the `ep.cuda.qmoe_row_tile_size` session configuration entry. - Keep `ORT_QMOE_ROW_TILE_SIZE` as a compatibility fallback. - Add optional QMoE diagnostics for route, tile size, workspace size, and tactic bucket. ## Qualification - End-to-end generation produced identical token output with tiling enabled. - In the qualified large MoE workload, a 64-row tile reduced QMoE temporary scratch from approximately 64 MB to 32 MB. Row tiling remains opt-in.
Author
Parents
Loading