DeepSpeed
Avoid zero-sized microbatches for incomplete minibatches when doing curriculum learning
#5118

Merged

Avoid zero-sized microbatches for incomplete minibatches when doing curriculum learning #5118

conglongli merged 19 commits into deepspeedai:master from avoid_zero_sized_microbatches_for_incomplete_minibatches

flush file before continuing. added assert

0fa6da65

removed if condition. collate_fn is by default None

4414e151

added assert of torch vs numpy types

14f2bbe1

removed 'data' key in favour or a global index

84d4398b

removed self.start_idx and self.end_idx

9a4cf512

metric_function as it was

7951289a

removed assert

8e6e556f

removed dtypes

00a9fed3

Merge branch 'bug_fix_flush_curriculum_reduce_file_concatenation'

82fbf19a

Merge branch 'remove_unnecessary_if_condition'

6ffad302

start and end indices

8c0ad6df

reverted 2 files that belong to a different PR

3a357b81

reverted 2 files that belong to a different PR

55fb0a74

bm-synth marked this pull request as ready for review 1 year ago

bm-synth requested a review from