transformers

[`fix`] Set input_modalities on various architectures that aren't just text

#44078

Merged