onnxruntime
fca94478 - [CUDA] Optimize CUDA prefill and quantized paged attention (#34117)

Commit
3 days ago
[CUDA] Optimize CUDA prefill and quantized paged attention (#34117) ### Description - Extend CUDA `VarlenCausalConvWithState` with dynamic grouped two-tap convolutions, including per-token coefficient deltas, boundary masking, and schema validation. - Add opt-in grouped BF16 INT3 MoE prefill, with a compatible SIMT path and guarded tensor-core acceleration using canonical recomputation for rounding-sensitive outputs. - Extend H128/GQA6 XQA support with BF16 INT8/INT4 speculative verification and BF16 INT4 single-token decoding. - Optimize INT8 KV prefill through sliding-window-aware gathering, vectorized dequantization, and revised FlashAttention query tiling. - Add BF16 shadow caching for global INT8 KV prefill. Retain a dequantized prefix across prefill chunks and convert only newly appended tokens, avoiding repeated full-prefix gathering. The optional `prefill_cache_control` input supplies an epoch, capacity, and final-chunk flag to explicitly govern reuse. - Add shadow-cache lifecycle safeguards: epoch invalidation, CUDA-stream synchronization, failure handling, and final-chunk buffer release. Missing controls, CUDA graph capture, multi-request batches, and unsupported configurations retain stateless behavior. - Expand correctness, dispatch, fallback, and cache-lifecycle coverage; update operator documentation and CUDA build dependencies. ### Motivation and Context There are existing bottlenecks in INT3 expert computation and repeated quantized-KV gathering, alongside gaps in BF16 quantized-cache speculative attention. These changes provide the CUDA primitives needed to reduce prefill overhead and support lower-memory speculative decoding. Optimizations are restricted to eligible configurations, with grouped INT3 prefill remaining opt-in. Prefix reuse requires explicit caller-managed lifetimes and trades temporary BF16 workspace for reduced dequantization work. The companion GenAI changes supply the necessary graph lowering and lifecycle controls. Model weights are not requantized; quantized-KV accuracy still requires application-specific qualification. ### Benchmarks Notes: 1. H200, batch 1, 512 output tokens, one warmup and five measured requests. The below benchmarks compare BF16 KV cache and INT8 KV cache versions of the RCs. TTFT and decode are medians; peak GPU memory includes weights and reserved pools, in decimal GB. INT8 KV cache versions use the optimized tile64 + shadow path; draft KV remains BF16 throughout. 2. Memory usage is not a fair comparison between ORT GenAI and llama.cpp because ORT GenAI tries to capture as much GPU memory as it can for future conversations. Memory usage is simply noted here for recording purposes. **INT3 RC** | Context | BF16 TTFT (s) | INT8 TTFT (s) | BF16 Decode (tok/s) | INT8 Decode (tok/s) | BF16 Peak (GB) | INT8 Peak (GB) | |---|---:|---:|---:|---:|---:|---:| | 8K* | 1.313 | 1.325 | 446.396 | **480.024** | 105.429 | 89.860 | | 16K* | 2.684 | 2.706 | 227.500 | **469.024** | 105.429 | 89.860 | | 32K* | 5.600 | 5.594 | 195.555 | **453.775** | 105.429 | 90.933 | | 64K | 12.157 | 12.041 | 302.338 | **428.886** | 105.429 | 92.141 | | 128K | 28.123 | 27.547 | 211.742 | **379.787** | 105.429 | 95.363 | **INT2 RC** | Context | BF16 TTFT (s) | INT8 TTFT (s) | BF16 Decode (tok/s) | INT8 Decode (tok/s) | BF16 Peak (GB) | INT8 Peak (GB) | |---|---:|---:|---:|---:|---:|---:| | 8K | 1.236 | 1.225 | 280.496 | **420.968** | 84.122 | 68.622 | | 16K | 2.490 | 2.502 | 405.926 | **441.012** | 84.122 | 68.620 | | 32K | 5.184 | 5.174 | 357.272 | **429.573** | 84.122 | 69.763 | | 64K | 11.297 | 11.127 | 275.144 | **391.317** | 84.122 | 70.834 | | 128K | 26.356 | 25.621 | 197.655 | **346.274** | 84.122 | 74.056 | **llama.cpp (2-bit)** | Prompt | TTFT (s) | Decode (tok/s) | Peak GPU (GB) | |---|---:|---:|---:| | 8K | 1.387 | 319.586 | 60.888 | | 16K | 2.839 | 330.673 | 61.250 | | 32K | 5.919 | 324.807 | 61.924 | | 64K | 13.035 | 317.357 | 63.268 | | 128K | 31.620 | 290.775 | 65.952 |
Parents
Loading