diffusers

86714b72 - Add unload_ip_adapter method (#6192)

Add unload_ip_adapter method (#6192) * Add unload_ip_adapter method * Update attn_processors with original layers * Add test * Use set_default_attn_processor --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>