DeepSpeed
Fix broadcast deadlock for incomplete batches in data sample for data analysis
#5117
Merged
Merged
Fix broadcast deadlock for incomplete batches in data sample for data analysis #5117
conglongli
merged
17 commits into
deepspeedai:master
from
fix_broadcast_deadlock_for_incomplete_batches
flush file before continuing. added assert
0fa6da65
removed if condition. collate_fn is by default None
4414e151
added assert of torch vs numpy types
14f2bbe1
removed 'data' key in favour or a global index
84d4398b
removed self.start_idx and self.end_idx
9a4cf512
metric_function as it was
7951289a
removed assert
8e6e556f
removed dtypes
00a9fed3
Merge branch 'bug_fix_flush_curriculum_reduce_file_concatenation'
82fbf19a
Merge branch 'remove_unnecessary_if_condition'
6ffad302
fixed broadcast deadlock
87d5db93
reverted 2 files that belong to a different PR
9c197d96
removed breakline
13d730cd
Merge branch 'master' into fix_broadcast_deadlock_for_incomplete_batches
6cf3053c
format
46aac6b3
Merge branch 'master' into fix_broadcast_deadlock_for_incomplete_batches
9864a4cc
Merge branch 'master' into fix_broadcast_deadlock_for_incomplete_batches
84d148a0
mrwyattii
approved these changes
on 2024-02-14
conglongli
approved these changes
on 2024-02-15
Login to write a write a comment.
Login via GitHub