Megatron-DeepSpeed
Alternative fix to TP > 1
#178
Merged

Alternative fix to TP > 1 #178

thomasw21
thomasw21 Only tp_rank_0 need to load dataset, we broadcast necessary meta data…
128c1588
thomasw21 thomasw21 requested a review from TevenLeScao TevenLeScao 4 years ago
thomasw21
thomasw21 commented on 2021-11-04
thomasw21 Collapse None and empty lists as well as empty dataloader and eval_it…
f3b64177
thomasw21 Woops
cf5cb451
TevenLeScao
TevenLeScao commented on 2021-11-04
thomasw21 Add plural to valid and test
9ced8c60
TevenLeScao
TevenLeScao TevenLeScao merged 2d9744f2 into main 4 years ago
thomasw21 thomasw21 deleted the thomas/alternative_fix_to_tp_over_1 branch 4 years ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
Milestone