diffusers

Fix model offload bug when key isn't present

#5030

Merged