Speed up the peft lora unload (#5741)
* Update peft_utils.py
* fix bug
* make the util backwards compatible.
Co-Authored-By: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
* fix import issue
* refactor the backward compatibilty condition
* rename the conditional variable
* address comments
Co-Authored-By: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
* address comment
---------
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
