[MLAS] RVV-Optimized LLM Operators for RISC-V (#28518)
## Description
Added RVV implementations for a subset of LLM inference operators.
Optimization of activation functions is in #28308.
All tests were conducted on a Spacemit K3 CPU (VLEN=256).
| Operator | File | Speedup vs Scalar | Precision |
| :--- | :--- | :--- | :--- |
| FP16 GEMM | `riscv64/halfgemm_kernel_rvv.cpp` | 51–191x | max_abs ≤
0.0005 (PASS) |
| FP16↔FP32 Cast | `riscv64/cast_kernel_rvv.cpp` | 4–12x | Bit-exact
(PASS) |
| RotaryEmbedding | `riscv64/rotary_embedding_kernel_rvv.cpp` | 3.1x |
max_abs ~6e-08 (PASS) |
| SimplifiedLayerNorm | `layer_norm_impl.cc` (inline RVV) | 4.3x |
Bit-exact (PASS) |
## Operator Performance
### **FP16 GEMM**
| Shape (M×N×K) | ORT Scalar | RVV | Speedup | Max Abs Error |
| :--- | :--- | :--- | :--- | :--- |
| 1×768×768 | 7.32 ms | 0.14 ms | 51.6x | 1.22e-04 |
| 32×768×768 | 235 ms | 1.25 ms | 187.9x | 3.66e-04 |
| 64×768×768 | 469 ms | 2.49 ms | 188.8x | 2.44e-04 |
| 128×3072×768 | 5718 ms | 30.4 ms | 187.8x | 4.88e-04 |
---
### **FP16↔FP32 Cast**
| Elements | Direction | ORT Scalar | RVV | Speedup |
| :--- | :--- | :--- | :--- | :--- |
| 1K | F16→F32 | 0.002 ms | 0.000 ms | 9.6x |
| 1K | F32→F16 | 0.002 ms | 0.000 ms | 10.5x |
| 64K | F16→F32 | 0.127 ms | 0.013 ms | 9.7x |
| 64K | F32→F16 | 0.150 ms | 0.013 ms | 11.3x |
| 1M | F16→F32 | 2.03 ms | 0.26 ms | 7.7x |
| 1M | F32→F16 | 2.39 ms | 0.50 ms | 4.8x |
---
### **RotaryEmbedding**
| Dim | Mode | ORT Scalar | RVV | Speedup | Max Abs Error |
| :--- | :--- | :--- | :--- | :--- | :--- |
| 64 | non-interleaved | 0.32 us | 0.05 us | 7.1x | 5.96e-08 |
| 64 | interleaved | 0.22 us | 0.07 us | 3.2x | 0 |
| 128 | non-interleaved | 0.64 us | 0.07 us | 9.7x | 5.96e-08 |
| 128 | interleaved | 0.44 us | 0.13 us | 3.4x | 0 |
| 256 | non-interleaved | 1.28 us | 0.10 us | 13.0x | 1.19e-07 |
| 256 | interleaved | 1.05 us | 0.25 us | 4.3x | 0 |
---
### **RMSNorm**
| Hidden | ORT Scalar | RVV | Speedup | Max Abs Error |
| :--- | :--- | :--- | :--- | :--- |
| 512 | 2.31 us | 0.38 us | 6.0x | 2.38e-07 |
| 1024 | 4.64 us | 0.71 us | 6.5x | 2.38e-07 |
| 2048 | 9.24 us | 1.42 us | 6.5x | 3.58e-07 |
| 4096 | 18.5 us | 2.82 us | 6.6x | 3.58e-06 |
> **Note**: ORT's LayerNorm ComputeJob is in an anonymous namespace —
there's no public API to call it separately. So I rewrite the benchmark
using the same algorithm as ORT's ComputeJob.
## Model Performance
The ONNX model comes from:
https://huggingface.co/onnx-community/Qwen3-0.6B-ONNX
| Metric | FP32 | FP16 |
| :--- | :--- | :--- |
| Prompt processing | 61.1 tok/s (255 ms p50) | 58.8 tok/s (272 ms p50)
|
| Token generation | 6.5 tok/s (152 ms p50) | 6.1 tok/s (162 ms p50) |
| E2E (16+32 tokens) | 4987 ms p50 | 5357 ms p50 |
| Peak memory | 3.1 GB | 4.1 GB |
> **Note**: FP32 is slightly faster because it runs SGEMM directly
without the FP16↔FP32 cast overhead. FP16 uses ~1 GB less storage on
disk but more runtime memory (the cast creates FP32 copies). Both use
the RVV SGEMM kernel for the actual compute.
