diffusers

Make sure we also change the config when setting `encoder_hid_dim_type=="text_proj"` and allow xformers

#3615

Merged