onnxruntime
4e24d372 - WIP: DP4AMatMul fix matmul for subgoup size 64 GPUs (#23637)

Commit
355 days ago
WIP: DP4AMatMul fix matmul for subgoup size 64 GPUs (#23637) ### Description This change moves away from using subgroup ops for quantization. This is because on AMD GPUs subgroup size is 64 and that is not handled in our quantization function, resulting in garbage output. Implementing subgroup size 64 quantization requires changing the workgroup size and then implementing support for subgroup size 128 becomes a challenge. With the new implementation perf on intel ALD remains about the same 4.36s for 1000K prefill. Tests for this change are present here https://github.com/microsoft/onnxruntime/blob/e66650350b85cb5e3a408f6576fe6a7f4f4ddebc/onnxruntime/test/contrib_ops/matmul_4bits_test.cc However, to trigger the current issue they must be run on a GPU with subgroup size 64.
Author
Parents
Loading