diffusers

[Refactor] refactor `loaders.py` to make it cleaner and leaner.

#5771

Merged