diffusers

Fix unloading of LoRAs when xformers attention procs are in use

#4179

Merged