diffusers

[chore] raise as early as possible in group offloading

#11792

Merged