diffusers

Remove device synchronization when loading weights

#11927

Merged