[Lazy init] Force fall back to slow init for composite models (#11705)
* fix encoder-decoder & RAG
* finalize
* Update src/transformers/models/encoder_decoder/modeling_encoder_decoder.py
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
* Update src/transformers/models/rag/modeling_rag.py
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
Co-authored-by: Patrick von Platen <patrick@huggingface.co>
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>