onnxruntime
9a43d1c9 - Preserve distinct epsilons in GQA pre-norm fusion (#33363)

Commit
6 days ago
Preserve distinct epsilons in GQA pre-norm fusion (#33363) ### Description Preserve separate Q and K normalization epsilons when fusing into GroupQueryAttention. Add optional `q_norm_epsilon` and `k_norm_epsilon` attributes, each falling back to the existing `qk_norm_epsilon`, and carry them through WebGPU and CUDA normalization paths. Equal-epsilon fusion retains the shared attribute; mismatched epsilons now fuse with both original values preserved. Update the generated attribute documentation and regression tests. ### Motivation and Context The fusion previously accepted Q/K epsilon differences up to 1e-9 and then applied Q's epsilon to K. Preserving both values fixes the numerical error while allowing fusion for all otherwise-supported positive finite epsilon pairs. Validation on Windows with fresh builds from this branch: | Build | Fusion tests | QK-normalization operator tests | | --- | --- | --- | | WebGPU | 21 passed, 4 CUDA-only skips | 5 passed, 6 CUDA-only skips | | CUDA 13.1 / cuDNN 9.19 | 21 passed, 4 WebGPU-only skips | 7 passed, 4 WebGPU-only skips | All 36 distinct focused tests passed across the two builds. New GPU fusion comparisons disable CPU fallback and check attention and cache outputs for decode and prefill. Tests cover independent overrides, the legacy fallback, invalid values, and stale attributes. The issue's attached model fuses with both epsilons preserved and matches all three unfused WebGPU outputs exactly (maximum absolute differences: 0, 0, 0). CUDA validation targeted SM120 with FlashAttention and memory-efficient attention disabled; those paths were not numerically validated. Changed-line clang-format 20.1.8, `git diff --check`, and the GQA attribute documentation check passed. No performance claim is made. Fixes #32930.
Author
Parents
Loading