DeepSpeed

970015bb - efficient communication layout for training MoE architecture at large scale