DeepSpeed

caba320a - fuse the all_to_all for the seq-parallel into one and use all_to_all_single