diffusers

[Tests] make test steps dependent on certain things and general cleanup of the workflows

#7026

Merged