onnxruntime
51df5b52 - Add packed attention workspace estimates (#32321)

Commit
73 days ago
Add packed attention workspace estimates (#32321) ## Summary Add MatMulNBits-equivalent operator-side workspace estimation for CUDA `PackedAttention` and `PackedMultiHeadAttention`: - Level 1 derives a conservative estimate from the node, resolved input shapes, CUDA device properties, and the EP's resolved attention options. - Level 2 declares the same estimate from positional `WorkspaceInputShape` metadata and constructed kernel state. - Existing graph-free runtime recipes remain the single source of truth for workspace bytes and layouts. - PackedAttention declares one 256-byte-aligned root in slot 0, with internal projection and attention regions. - PackedMultiHeadAttention declares one 256-byte-aligned attention root in slot 0. Level 1 is log-only, matching the current MatMulNBits pilot. This PR does not add #32071-specific planner APIs or change runtime `GetScratchBuffer()` behavior. ## Route aggregation Runtime routes are mutually exclusive, so the estimate uses: ```text PackedAttention: align_up(projection_bytes, 256) + max(feasible TRT, MEA, unfused recipes) PackedMultiHeadAttention: max(feasible Flash, TRT, MEA, unfused recipes) ``` Route reachability is evaluated conservatively for every runtime shape up to the supplied maximum geometry. This is necessary because Flash/MEA thresholds and attention-bias alignment gates are not monotonic when moving from a maximum shape to a smaller runtime shape. Unfused fallback is always retained, and a failure to size any included route makes the estimate unavailable rather than silently underestimating. ## Shape and zero semantics - Missing mandatory inputs, shapeless required inputs, unknown dimensions, malformed geometry, and checked-arithmetic overflow produce no estimate. - `WorkspaceInputShape` does not carry max-shape provenance, so zero-shaped framework hints are conservatively treated as unavailable. - Exact zero behavior remains supported by the graph-free runtime recipes. - At the current Level-2 boundary, both unavailable and zero are represented by an empty requirements list. ## Planner integration - Both operators fit #32071's current one-slot pilot. Generic framework multi-slot support remains unchanged. - A declaration alone is not planner opt-in. `SupportsPreallocatedWorkspace()`, slot-0 retrieval, and PA root slicing must land atomically in the planner integration. - Until then, PA retains its two dynamic allocations and PMHA retains its one dynamic allocation. ## Build boundaries The framework adapters and kernel overrides are excluded from: - CUDA minimal builds - `DISABLE_CONTRIB_OPS` builds - CUDA plugin EP builds The graph-free workspace recipes remain available to the shared BERT attention infrastructure where required. ## Validation - 18/18 PA/PMHA workspace estimator tests - includes direct production-kernel Level-2 declaration tests - route-threshold, max-not-sum, aligned-root padding/no-padding, optional-hole, zero, overflow, and malformed geometry coverage - 23/23 existing packed-attention workspace recipe tests - 20/20 existing hand-calculated runtime parity cases - 26/26 PackedAttention/PackedMultiHeadAttention runtime operator tests - CUDA provider-test build - 145 CUDA internal tests executed: 143 passed, 2 unrelated LeanAttention skips - `DISABLE_CONTRIB_OPS` and CUDA-minimal compile-guard probes - C++ formatting and diff checks ## Dependency This is a stacked follow-up to #32312. The base should change to `main` after #32312 merges. Tracking: #29775 --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Copilot-Session: c04148cc-7ace-4cf4-b981-0ddc92334e78
Author
Parents
Loading