onnxruntime
5b021b24 - [webgpu] Add support for Q2 in matmulnbits (#25763)

Commit
1 year ago
[webgpu] Add support for Q2 in matmulnbits (#25763) ### Description This change adds support for Q2 quantized matmulnbits, in webgpu. ### Motivation and Context An alternate way to support bitnets is through adding support for lower bits in matmulnbits, this reuses our shaders and is more maintainable than a separate op. The model size grows a bit however for a 2B parameter model using 1.58bpw vs 2bpw the size difference is just 100MB. The simpler dequantization also improves perf, on an Intel XE matmul looks to be 20% faster using q2 weights vs q4 weights for the same matrix dimensions. Q2 version of the bitnet model is here https://huggingface.co/sushraja/bitnet-b1.58-2B-4T-fp16-onnx/tree/main/bitnet_q2
Author
Parents
Loading