diffusers

Fix: leaf_level offloading breaks after delete_adapters

#12639

Merged