[ip-adapter] refactor `prepare_ip_adapter_image_embeds` and skip load `image_encoder` (#7016)
* add
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
---------
Co-authored-by: yiyixuxu <yixu310@gmail,com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>