Megatron-DeepSpeed

2d9744f2 - Alternative fix to TP > 1 (#178)

Alternative fix to TP > 1 (#178) * Only tp_rank_0 need to load dataset, we broadcast necessary meta data to the other ranks * Collapse None and empty lists as well as empty dataloader and eval_iters ==0 * Woops * Add plural to valid and test