[`core` / `PEFT` / `LoRA`] Integrate PEFT into Unet (#5151)
* v1
* add tests and fix previous failing tests
* fix CI
* add tests + v1 `PeftLayerScaler`
* style
* add scale retrieving mechanism system
* fix CI
* up
* up
* simple approach --> not same results for some reason
* fix issues
* fix copies
* remove unneeded method
* active adapters!
* fix merge conflicts
* up
* up
* kohya - test-1
* Apply suggestions from code review
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* fix scale
* fix copies
* add comment
* multi adapters
* fix tests
* oops
* v1 faster loading - in progress
* Revert "v1 faster loading - in progress"
This reverts commit ac925f81321e95fc8168184c3346bf3d75404d5a.
* kohya same generation
* fix some slow tests
* peft integration features for unet lora
1. Support for Multiple ranks/alphas
2. Support for Multiple active adapters
3. Support for enabling/disabling LoRAs
* fix `get_peft_kwargs`
* Update loaders.py
* add some tests
* add unfuse tests
* fix tests
* up
* add set adapter from sourab and tests
* fix multi adapter tests
* style & quality
* style
* remove comment
* fix `adapter_name` issues
* fix unet adapter name for sdxl
* fix enabling/disabling adapters
* fix fuse / unfuse unet
* nit
* fix
* up
* fix cpu offloading
* fix another slow test
* fix another offload test
* add more tests
* all slow tests pass
* style
* fix alpha pattern for unet and text encoder
* Update src/diffusers/loaders.py
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
* Update src/diffusers/models/attention.py
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
* up
* up
* clarify comment
* comments
* change comment order
* change comment order
* stylr & quality
* Update tests/lora/test_lora_layers_peft.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* fix bugs and add tests
* Update src/diffusers/models/modeling_utils.py
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
* Update src/diffusers/models/modeling_utils.py
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
* refactor
* suggestion
* add break statemebt
* add compile tests
* move slow tests to peft tests as I modified them
* quality
* refactor a bit
* style
* change import
* style
* fix CI
* refactor slow tests one last time
* style
* oops
* oops
* oops
* final tweak tests
* Apply suggestions from code review
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* Update src/diffusers/loaders.py
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* comments
* Apply suggestions from code review
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* remove comments
* more comments
* try
* revert
* add `safe_merge` tests
* add comment
* style, comments and run tests in fp16
* add warnings
* fix doc test
* replace with `adapter_weights`
* add `get_active_adapters()`
* expose `get_list_adapters` method
* better error message
* Apply suggestions from code review
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* style
* trigger slow lora tests
* fix tests
* maybe fix last test
* revert
* Update src/diffusers/loaders.py
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
* Update src/diffusers/loaders.py
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
* Update src/diffusers/loaders.py
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
* Update src/diffusers/loaders.py
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* move `MIN_PEFT_VERSION`
* Apply suggestions from code review
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* let's not use class variable
* fix few nits
* change a bit offloading logic
* check earlier
* rm unneeded block
* break long line
* return empty list
* change logic a bit and address comments
* add typehint
* remove parenthesis
* fix
* revert to fp16 in tests
* add to gpu
* revert to old test
* style
* Update src/diffusers/loaders.py
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
* change indent
* Apply suggestions from code review
* Apply suggestions from code review
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Sourab Mangrulkar <13534540+pacman100@users.noreply.github.com>
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
