Fix missing initializations for models created in 2023 (#39239)
* fix SwiftFormer
* fix Kosmos2
* fix Owlv2
* fix Sam
* fix Vits
* fix Pvt
* fix MobileViTV2
* fix PatchTST
* fix Bros
* fix Informer
* fix BridgeTower
* fix Mra and Yoso
* fix Rwkv
* fix EfficientNet
* fix NllbMoe
* fix Tvp
* fix Clap
* fix Autoformer
* fix SwiftFormer
* fix Mgpstr
* fix Align
* fix VitMatte
* fix SpeechT5
* add conditional check for parameters
* fix SpeechT5
* fix TimmBackbone and Clvp
* fix SwiftFormer
* fix SeamlessM4T and SeamlessM4Tv2
* fix Align
* fix Owlv2 and OwlViT
* add reviewed changes
* add reviewed changes
* fix typo
---------
Co-authored-by: Cyril Vallez <cyril.vallez@huggingface.co>