[LoRA] don't break offloading for incompatible lora ckpts. (#5085)
* don't break offloading for incompatible lora ckpts.
* debugging
* better condition.
* fix
* fix
* fix
* fix
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>