onnxruntime
af0a4bab - [WebGPU] Batch streamless GPU-to-GPU copy vectors (#33567)

Commit
5 days ago
[WebGPU] Batch streamless GPU-to-GPU copy vectors (#33567) ### Description <!-- Describe your changes. --> Batch compatible streamless GPU-to-GPU `OrtApi::CopyTensors` vectors into one call-local recording and queue submission by default. - Encode copies in array order, preserving dependencies between entries. - Retain the context through submission and isolate recordings for concurrent calls and error recovery. - Preserve existing paths for single copies, CPU transfers, explicit streams, and incompatible vectors. - Add provider tests for submission counts, stream routing, copy dependencies, error recovery, buffer lifetimes, and concurrent Session teardown. No public API/ABI changes or experimental flags are introduced. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Fixed-state caching can issue many small GPU copies during each execution phase. Submitting every copy separately adds avoidable host and queue-submission overhead. This change allows callers to submit those copies together without adding a host-completion wait. The companion consumer is microsoft/onnxruntime-genai#2775. Together, the changes saved approximately 0.92 seconds for baseline generation and 1.52 seconds for DFlash2 width 3 in isolated A100 measurements with 512 generated tokens. These savings are configuration-dependent; generated tokens, speculative counters, and final fixed-state bytes were unchanged. Validation: Release runtime/plugin builds passed. The expanded WebGPU provider suites passed 66 tests, and serialized compatibility passed 24 tests, with no skips and two pre-existing disabled tests. Formatting/lint checks passed. Local execution used Linux/Vulkan on an A100; other backends remain for platform CI. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Author
Parents
Loading