transformers
598d8ba8 - Resolve continuous batching config against the text config for composite models (#48299)

Commit
2 days ago
Resolve continuous batching config against the text config for composite models (#48299) PagedAttentionCache and the CB config resolution read decoder attributes (num_key_value_heads, num_hidden_layers, head_dim, ...) from the top-level config, which does not carry them for composite configs (vision-language models keep them on text_config). Continuous batching on any ForConditionalGeneration model crashed with 'num_key_value_heads or num_attention_heads could not be found in the config'. Resolve against config.get_text_config(), a no-op for text-only configs.
Author
Parents
Loading