diffusers

Fix loading broken LoRAs that could give NaN

#5316

Merged