onnxruntime
d8024cdc - Improve CUDA RMSNorm reduction and aligned half2 output (#33178)

Commit
29 days ago
Improve CUDA RMSNorm reduction and aligned half2 output (#33178) ### Description - Remove unused mean/count reduction work from simplified normalization and use FP32 arithmetic for paired FP16 output. - Require actual input, scale, and output half2 alignment in addition to even dimensions/offsets; retain scalar fallback for offset external buffers. - Remove the unused half helper and correct double rsqrt dispatch. - Add documented dtype/broadcast/range tests and CUDA IO-binding regressions for aligned and independently offset buffers. ### Motivation and Context Even tensor dimensions do not guarantee aligned external buffers. A two-byte-offset FP16 CUDA input previously caused CUDA error 716; the address checks prevent that invalid half2 access without disabling CUDA graphs. ### Validation Standalone Release CPU/CUDA build of this branch on Linux/A100 (SM80): 94 native tests passed; the focused CUDA IO-binding test passed all 10 subcases with CPU fallback disabled. Changed-file lint and whitespace checks passed. No full-suite, Windows, ROCm, or other GPU-architecture validation. The alignment guard was also measured within the combined Gemma 4 runtime: CUDA graphs and INT4 lm_head retained, identical outputs across 21 full-model runs. Fixed-provider decode measured 215.13 +/- 1.21 tokens/s versus 215.75 +/- 0.71 for the pooled before-provider runs (-0.29%). This small observed difference is comparable to run variation, not proof of zero impact on all workloads. The standalone draft was not separately full-model benchmarked. This is one of four independent ORT drafts; no other draft commits were included in its native build.
Author
Parents
Loading