diffusers

[Core] enable lora for sdxl controlnets too and add slow tests.

#4666

Merged