llama.cpp
ace20f67 - graph: pad the KQ bias of the no-cache attention (#30446)

Commit
25 days ago
graph: pad the KQ bias of the no-cache attention (#30446) The no-cache attention pads K, V and the mask to 256, but T5 also adds a relative position bias of [n_tokens, n_tokens], so the add with KQ fails unless n_tokens divides the padded width. Pad the bias with zeros up to the KV dimension of the mask, like K and V.
Parents
Loading