KV cache quantization for ORT WebGPU (#28059)
### Description
Turbo quant implementation for ORT WebGPU, using a Hadamard matrix for
rotation instead of a regular matrix, which deviates from the paper.
| seq_length | turbo_quant | prefill_tps | generation_tps |
working_set_gb | gpu_memory_gb | % saving|
|-----------|-------------|-------------|----------------|----------------|---------------|----------------|
| 1024 | ❌ Off| 2007.21 | 108.48 | 1.84 | 4.14 | |
| | ✅ On| 2053.70 | 113.06 | 1.64 | 3.37 | **18.6%** |
| 2048 | ❌ Off | 1778.32 | 111.26 | 1.95 | 5.49 | |
| | ✅ On | 1763.69 | 111.83 | 1.90 | 3.75 | **31.7%** |
| 4096 |❌ Off | 1373.88 | 29.78 | 1.61 | 7.21 | |
| |✅ On | 1367.89 | 29.51 | 2.30 | 4.96 | **31.2%** |
| 8192 | ❌ Off | 948.90 | 84.17 | 2.44 | 10.14 | |
| | ✅ On | 943.96 | 82.36 | 2.31 | 6.50 | **35.9%** |
- Hadamard transform is kept as its own class and standalone shader
-[hadamard_transform.h](https://github.com/microsoft/onnxruntime/pull/28059/changes#diff-eb3f846a5a284367f10b2059cc662e19d1b029b416c0c8373e20add9dbfe8afa)
- used to rotate/unrotate Q. Can be used by other feature in the future
like activation quantization.
- TurboQuantHadamard applies Hadamard transform and then quantizes using
the centroid look up for q4.
- Dequantization is all fused into the various flash attention kernels.
Caller for LLMs like gen-ai have to set kvCacheQuantizationBits:4 in the
EP provider options and pass in present,past kv cache input, output
tensors that have a reduced headsize.
With turboquant the headsize reduces from say using 16bits per value to
4bits and in addition there is a 32bit scale in the front of each token
per head.
----------------------------------------------------------------
Note on impact on quality. Evaluting KV quantization 4 bits with Phi4
mini
| Metric | kv0 (no quant) | kv4 (4-bit) | Δ (kv4 − kv0) |
|---|---:|---:|---:|
| **Mean quality score (0–5)** | **3.64** | **3.36** | **−0.28** |
| Head-to-head wins | **71** | 37 | — |
| Ties | — | — | 92 |
| Broken/failed (score ≤ 1) | 13 | **17** | +4 |
| Hard-broken (score = 0) | 2 | 2 | 0 |
**Verdict:** Under graded rubric scoring, **4-bit KV quantization shows
a small but consistent quality penalty** (≈ −0.28 on a 5-point scale).
kv0 wins roughly 2× as many head-to-head matchups as kv4 (71 vs 37),
though nearly half of all prompts (92/200) tie. The degradation is
**mild and uneven**, not catastrophic — it concentrates in specific use
cases rather than degrading everything.
## Quality-score distribution
| Score | kv0 | kv4 |
|---:|---:|---:|
| 5 (perfect) | **62** | 42 |
| 4 (good) | 56 | 55 |
| 3 (bearable) | 44 | 54 |
| 2 (significant issues) | 25 | 32 |
| 1 (serious issues) | 11 | 15 |
| 0 (broken) | 2 | 2 |
The main shift is at the **top end**: kv0 earns 62 perfect scores vs
kv4's 42 (−20). Those lost 5s mostly slide down to 3s (+10) and 2s (+7).
kv4 doesn't produce dramatically more total failures — it produces fewer
*flawless* answers.
4-bit KV quant is acceptable for latency/memory-sensitive deployments
where a ~0.3-point average quality dip is tolerable, **except** for
tag-generation and content-detection workloads, where kv0 (no quant) is
meaningfully better. If those use cases matter, keep KV quant off or
pair it with `repetition_penalty > 1.0` to suppress the tag-loop
failures that dominate kv4's losses.
