Enable contiguous gradients with Z1+MoE

Commit

3 years ago

Enable contiguous gradients with Z1+MoE MoE training with zero stage 1 only works with `contiguous gradients=True`.