transformers
6096273e - Skip the query/key head repeat for grouped value heads in gated DeltaNet (#49141)

Commit
7 days ago
Skip the query/key head repeat for grouped value heads in gated DeltaNet (#49141) * Skip the query/key head repeat for grouped value heads in gated DeltaNet FLA's chunked and fused recurrent gated delta rule kernels accept fewer query/key heads than value heads since 0.5.0, and the hub kernel does too. The torch fallbacks now repeat query/key themselves, so the modeling code only repeats when an older FLA is installed. This removes a query/key copy and its backward reduction in every linear-attention layer of Qwen3-Next, Qwen3.5, Qwen3.5-MoE, Qwen4-Exp and OLMo-Hybrid. * Pass g and beta by keyword in test_gdn_chunked_matches_recurrent FLA's fused recurrent kernel takes gk and gv between g and beta, so the positional call fed beta into gk whenever FLA was installed. FLA's chunked kernel also runs its dot products in TF32, so the tolerance follows the backend. * Name the FLA version check fla_needs_equal_qk_heads * Always pass grouped query/key heads to the gated delta rule kernels Drop the FLA version check and the layer-level repeat; the torch fallbacks repeat query/key themselves when there are fewer of them. * Add a helper for the gated delta rule test inputs * Move the gated delta rule input helper into the model tester
Author
Parents
Loading