[Single File Loading] Handle unexpected keys in CLIP models when `accelerate` isn't installed. (#8462)
* update
* update
* update
* update
* update
---------
Co-authored-by: YiYi Xu <yixu310@gmail.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>