[LoRA] make set_adapters() method more robust. (#9535)
* make set_adapters() method more robust.
* remove patch
* better and concise code.
* Update src/diffusers/loaders/lora_base.py
Co-authored-by: YiYi Xu <yixu310@gmail.com>
---------
Co-authored-by: YiYi Xu <yixu310@gmail.com>
