Megatron-DeepSpeed

f75af1f9 - Offload to CPU earlier & increase number of bs in pipleine parallelism