[SD3 LoRA Training] Fix errors when not training text encoders (#8743)
* fix
* fix things.
Co-authored-by: Linoy Tsaban <linoy.tsaban@gmail.com>
* remove patch
* apply suggestions
---------
Co-authored-by: Linoy Tsaban <57615435+linoytsaban@users.noreply.github.com>
Co-authored-by: sayakpaul <spsayakpaul@gmail.com>
Co-authored-by: Linoy Tsaban <linoy.tsaban@gmail.com>