[Core] LoRA improvements pt. 3 (#4842)
* throw warning when more than one lora is attempted to be fused.
* introduce support of lora scale during fusion.
* change test name
* changes
* change to _lora_scale
* lora_scale to call whenever applicable.
* debugging
* lora_scale additional.
* cross_attention_kwargs
* lora_scale -> scale.
* lora_scale fix
* lora_scale in patched projection.
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* styling.
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* remove unneeded prints.
* remove unneeded prints.
* assign cross_attention_kwargs.
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* debugging
* clean up.
* refactor scale retrieval logic a bit.
* fix nonetypw
* fix: tests
* add more tests
* more fixes.
* figure out a way to pass lora_scale.
* Apply suggestions from code review
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* unify the retrieval logic of lora_scale.
* move adjust_lora_scale_text_encoder to lora.py.
* introduce dynamic adjustment lora scale support to sd
* fix up copies
* Empty-Commit
* add: test to check fusion equivalence on different scales.
* handle lora fusion warning.
* make lora smaller
* make lora smaller
* make lora smaller
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
