[CUDA] Optimize CUDA prefill and quantized paged attention (#34117)
### Description
- Extend CUDA `VarlenCausalConvWithState` with dynamic grouped two-tap
convolutions, including per-token coefficient deltas, boundary masking,
and schema validation.
- Add opt-in grouped BF16 INT3 MoE prefill, with a compatible SIMT path
and guarded tensor-core acceleration using canonical recomputation for
rounding-sensitive outputs.
- Extend H128/GQA6 XQA support with BF16 INT8/INT4 speculative
verification and BF16 INT4 single-token decoding.
- Optimize INT8 KV prefill through sliding-window-aware gathering,
vectorized dequantization, and revised FlashAttention query tiling.
- Add BF16 shadow caching for global INT8 KV prefill. Retain a
dequantized prefix across prefill chunks and convert only newly appended
tokens, avoiding repeated full-prefix gathering. The optional
`prefill_cache_control` input supplies an epoch, capacity, and
final-chunk flag to explicitly govern reuse.
- Add shadow-cache lifecycle safeguards: epoch invalidation, CUDA-stream
synchronization, failure handling, and final-chunk buffer release.
Missing controls, CUDA graph capture, multi-request batches, and
unsupported configurations retain stateless behavior.
- Expand correctness, dispatch, fallback, and cache-lifecycle coverage;
update operator documentation and CUDA build dependencies.
### Motivation and Context
There are existing bottlenecks in INT3 expert computation and repeated
quantized-KV gathering, alongside gaps in BF16 quantized-cache
speculative attention. These changes provide the CUDA primitives needed
to reduce prefill overhead and support lower-memory speculative
decoding.
Optimizations are restricted to eligible configurations, with grouped
INT3 prefill remaining opt-in. Prefix reuse requires explicit
caller-managed lifetimes and trades temporary BF16 workspace for reduced
dequantization work. The companion GenAI changes supply the necessary
graph lowering and lifecycle controls. Model weights are not
requantized; quantized-KV accuracy still requires application-specific
qualification.
### Benchmarks
Notes:
1. H200, batch 1, 512 output tokens, one warmup and five measured
requests. The below benchmarks compare BF16 KV cache and INT8 KV cache
versions of the RCs. TTFT and decode are medians; peak GPU memory
includes weights and reserved pools, in decimal GB. INT8 KV cache
versions use the optimized tile64 + shadow path; draft KV remains BF16
throughout.
2. Memory usage is not a fair comparison between ORT GenAI and llama.cpp
because ORT GenAI tries to capture as much GPU memory as it can for
future conversations. Memory usage is simply noted here for recording
purposes.
**INT3 RC**
| Context | BF16 TTFT (s) | INT8 TTFT (s) | BF16 Decode (tok/s) | INT8
Decode (tok/s) | BF16 Peak (GB) | INT8 Peak (GB) |
|---|---:|---:|---:|---:|---:|---:|
| 8K* | 1.313 | 1.325 | 446.396 | **480.024** | 105.429 | 89.860 |
| 16K* | 2.684 | 2.706 | 227.500 | **469.024** | 105.429 | 89.860 |
| 32K* | 5.600 | 5.594 | 195.555 | **453.775** | 105.429 | 90.933 |
| 64K | 12.157 | 12.041 | 302.338 | **428.886** | 105.429 | 92.141 |
| 128K | 28.123 | 27.547 | 211.742 | **379.787** | 105.429 | 95.363 |
**INT2 RC**
| Context | BF16 TTFT (s) | INT8 TTFT (s) | BF16 Decode (tok/s) | INT8
Decode (tok/s) | BF16 Peak (GB) | INT8 Peak (GB) |
|---|---:|---:|---:|---:|---:|---:|
| 8K | 1.236 | 1.225 | 280.496 | **420.968** | 84.122 | 68.622 |
| 16K | 2.490 | 2.502 | 405.926 | **441.012** | 84.122 | 68.620 |
| 32K | 5.184 | 5.174 | 357.272 | **429.573** | 84.122 | 69.763 |
| 64K | 11.297 | 11.127 | 275.144 | **391.317** | 84.122 | 70.834 |
| 128K | 26.356 | 25.621 | 197.655 | **346.274** | 84.122 | 74.056 |
**llama.cpp (2-bit)**
| Prompt | TTFT (s) | Decode (tok/s) | Peak GPU (GB) |
|---|---:|---:|---:|
| 8K | 1.387 | 319.586 | 60.888 |
| 16K | 2.839 | 330.673 | 61.250 |
| 32K | 5.919 | 324.807 | 61.924 |
| 64K | 13.035 | 317.357 | 63.268 |
| 128K | 31.620 | 290.775 | 65.952 |