onnxruntime
327d11ab - [CUDA] Bound GQA non-windowed workspace via the KV-length envelope (route-aware L1 estimate) (#33016)

Commit
31 days ago
[CUDA] Bound GQA non-windowed workspace via the KV-length envelope (route-aware L1 estimate) (#33016) ## Summary Adds opt-in, route-aware Level-1 workspace estimation for non-windowed CUDA GroupQueryAttention using the declared total-KV envelope from `ep.cuda.gqa_workspace_max_total_sequence_length`. The present-cache capacity bound is **`max(allocated past-cache capacity, declared total-KV envelope)`**. Static caches may be larger than the active sequence: GQA permits `total_sequence_length <= past_sequence_length` and retains that allocated capacity. Bounding only the envelope would underestimate MEA expansion and partial-alias preservation. The existing graph-free aggregate sizes reachable routes and admitted prompt/decode phase corners; this PR does not introduce a second sizing engine. ## Envelope validation and caller contract - Explicit values are parsed once during session initialization and must be decimal integers in **`[0, INT32_MAX]`**, matching the runtime ABI. Malformed, negative, overflowing, and greater-than-`INT32_MAX` values fail initialization. - Zero remains the unspecified value. With the envelope unset or zero, non-windowed estimation is unavailable; windowed estimation continues to use its cache-shape bound. - The envelope must cover the query-length bound, and the resulting cache-capacity bound must fit the runtime int32 ABI. - The query/max-shape and total-KV declarations are estimation inputs, **not runtime-enforced admission limits or a no-OOM guarantee**. The proof applies only within the caller-declared workload envelope; runtime enforcement remains follow-up work. ## Workspace accounting and proof policy - Capacity-sensitive preparation and MEA expansion cover the full static-cache extent, even when it exceeds the active total-KV envelope. - Partial aliasing copies the full past tensor. Its preservation scratch coexists with the chosen route and is added above the per-route maximum, bounded by the full cache-capacity bound. - Preparation recipes already account for sequence-length vectors; there is no new missing-vector gap. - Reachable cuDNN without a workspace oracle, unsupported attention-bias cases, and otherwise unprovable route domains conservatively make the estimator unavailable. - For capacity-aware CUDA placement, every GQA candidate must have a successful analytical estimate. If proof is unavailable, CUDA assignment for that node is declined. - The generic accountant fallback is an ad hoc safety margin, not a proven GQA bound. Profile data is empirical and likewise cannot substitute for the missing analytical proof. Unrelated operators retain their existing generic fallback behavior. ## Regression coverage - Accepted boundary value `INT32_MAX`; rejected `INT32_MAX + 1`, `INT64_MAX`, decimal overflow, malformed text, and negative values. - Unset-envelope behavior and monotonicity when the envelope increases. - Static past capacity **1024** with total-KV envelope **512**, checking full preservation and MEA expansion against complete concrete runtime recipes. - An envelope below the query-length bound still declines, even when the static cache is larger. - Static capacity outside the runtime int32 ABI declines. - Parsed-envelope plumbing and unchanged windowed estimates. - Additive full-past preservation above the per-route maximum. - Capacity-aware CUDA placement declines GQA when the analytical estimator is unavailable, including when profile data exists; unrelated operators retain generic fallback. ## Dependencies Targets `main`, which now includes #33118's shared preparation-sizing implementation. The duplicated sizing commits were dropped during the post-merge rebase; this PR contains only its envelope-estimation, hardening, test, and documentation changes. Part of the policy-based workspace-estimation series described in #32944. Committed per-`(node, phase)` route policy and runtime workload-envelope enforcement remain follow-up work. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Chi Lo <lochi@microsoft.com> Copilot-Session: 7fa98d07-aec3-41b7-92b8-b0ec317a95bf Copilot-Session: c132130a-b0c7-422c-9ff7-75826b6ea3fc Copilot-Session: b95559c4-5cbe-4787-8dd7-ffa45a2157f9
Author
Parents
Loading