onnxruntime
b016c164 - [CUDA] Optimize packed QSA scoring and multi-token verification (#33328)

Commit
30 days ago
[CUDA] Optimize packed QSA scoring and multi-token verification (#33328) ### Description Adds targeted optimizations to CUDA `PackedSparseAttentionIndexer`: - Bound hierarchical scoring work using device-resident causal and state lengths. - Reuse key vectors across heads and verification queries in eligible single-request calls. - Compact candidate lists before subsequent hierarchical merging. - Add regression coverage for ragged batches, empty requests, and multi-query verification across FP32, FP16, and BF16. - Extend the existing device-source probe with QSA parity checks and score/merge timing modes. The changes retain query-specific visibility and stable selection ordering without changing the operator schema or state ABI. ### Motivation + Context Packed QSA can reserve substantially more state capacity than a query can causally access. Scoring unused capacity and repeatedly processing full candidate lists creates unnecessary work. Multi-token verification also lets multiple queries share key loads. Device-side bounds, query-paired reuse, and candidate compaction reduce redundant scoring and merging without host metadata readback. This PR is stacked on [#33094](https://github.com/microsoft/onnxruntime/pull/33094). NVFP4 GEMV changes and their probes belong to that lower PR and are not part of this PR’s diff. --------- Co-authored-by: Akshay Sonawane <asonawane@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Sayan Shaw <52221015+sayanshaw24@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Parents
Loading