[WIP] FIX Make Mixtral LoRA loading work (#44478)
* [WIP] FIX Make Mixtral LoRA loading work
Required fixes:
- some code was using unordered data structures, making weight order
random
- adjust alpha to offset increased rank from fusion
- import functions from PEFT if available
* Expose a function to apply weight conversion
This can be used in PEFT to apply weight conversion there without having
to either re-implement the whole weight conversion machinery or having
to call transformer_model.load_adapter. We want to avoid the latter
because there is a lot of PEFT custom logic to weight loading which
would require adjusting for that case.
* Move more stuff to PEFT
mark stuff that can be removed with peft 0.19
* Skip test unless PEFT >= 0.19
* Reviewer feedback: fix import error
---------
Co-authored-by: Cyril Vallez <cyril.vallez@huggingface.co>
