onnxruntime
7fe62a35 - [CUDA] Add compact GatedDeltaNet operator (#32282)

Commit
9 days ago
[CUDA] Add compact GatedDeltaNet operator (#32282) ### Description - Add the CUDA `GatedDeltaNet` contrib operator with recurrent decode and tensor-core chunked prefill paths. - Support ragged and rank-4 inputs, fused Qwen gate normalization, and native GDN arithmetic from raw `A_log`. - Return the compact three-output ABI `[output, final_state, state_update]`. The FP32 `state_update` capsule packs decay, shared-key, and delta transitions needed to replay accepted speculative prefixes without materializing dense recurrent checkpoints. - Add schema and shape validation, CUDA registration, focused tests, a microbenchmark, and an authored operator guide. ### Motivation and Context Dense recurrent checkpoints scale the full FP32 state with draft width. At the Qwen3.8 geometry, a four-slot window across 48 GDN layers consumes 576 MiB. Compact transition capture keeps one committed state and records only the information needed to reconstruct an accepted prefix. The schema intentionally exposes only native arithmetic and does not include the experimental `arithmetic_mode` attribute. Models exported with the retired experimental ABI must be re-exported. Companion ONNX Runtime GenAI integration: https://github.com/microsoft/onnxruntime-genai/pull/2472 ### Validation - `./build/cu130/Debug/onnxruntime_provider_test --gtest_filter='*GatedDeltaNet*'` - 26/26 focused tests passed on H200 with CUDA 13.0 and cuDNN 9.23. ### Performance and Quality On H200, context 2048, generation 256, with five paired fresh-process repetitions per batch: - Native arithmetic won all 40 MTP and DFlash2 throughput pairs versus the retired compatibility experiment. Median native/compatibility ratios were 1.0519 for MTP and 1.0543 for DFlash2. - Paired quality differences were not statistically significant: MMLU-Pro 83.75% native vs. 83.38% compatibility (McNemar p=0.73); GPQA 81.82% vs. 78.28% (p=0.23). - Against the retired separate-factor representation, the packed capsule preserved exact tokens and replay work. It won 20/20 MTP pairs and was at parity for DFlash2 batch 16, with a small-batch benefit.
Author
Parents
Loading