diffusers

Make sure torch compile doesn't access unet config

#4008

Merged