Fix loading broken LoRAs that could give NaN (#5316)
* Fix fuse Lora
* improve a bit
* make style
* Update src/diffusers/models/lora.py
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
* ciao C file
* ciao C file
* test & make style
---------
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
