onnxruntime
0d291bc5 - Enable split-KV for paged FlashAttention decode (#32102)

Commit
2 days ago
Enable split-KV for paged FlashAttention decode (#32102) ### Summary Enable`FlashAttention`'s existing split-KV path for paged decode. Split-KV divides long KV sequences across multiple CUDA thread blocks and combines their partial results. This improves GPU utilization during low-batch, long-context decoding. The optimization is limited to safe cases: - decode with one new token per request; - global attention without attention sinks; - KV sequences longer than 512 tokens. Short contexts, local-window attention, prefill, and mixed-length query batches retain their existing paths.
Author
Parents
Loading