diffusers

Fix potential type mismatch errors in SDXL pipelines

#4796

Merged