llama.cpp
99f2141b - metal : support BF16 src1 without the tensor API

Commit
2 hours ago
metal : support BF16 src1 without the tensor API Allow GGML_PREC_BF16 when the device has bfloat, not only when it has the Metal4 tensor API. Add legacy _prec_bf16 kernels for MUL_MAT and MUL_MAT_ID that stage src1 as bfloat in the threadgroup and run the simdgroup multiply with bfloat A and B. The F16 MUL_MAT_ID kernel uses half A on the tensor path and bfloat A on the legacy path. Cast the quantized dequantize outputs to the destination element type: the older Metal compiler on M2 rejects implicit float to bfloat. Tested on M2 Ultra (legacy): MUL_MAT_PREC_SRC1 24/24 and MUL_MAT_ID_PREC_SRC1_BF16 30/30. Flint Q4_K_M with LLAMA_MOE_AMAX_SKIP=1: pp512 -2.1%, tg128 +25.5% vs master. On M5 Max: pp512 +2.1%, tg128 +22.2%. Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
Author
Committer
Parents
Loading