PR #82 update merge_preprocessed_data to use distributed merge

update merge_preprocessed_data to use parallel merge

adammoody committed 4 years ago

indexed_dataset: add docstrings to merge and gather methods

adammoody committed 4 years ago

merge_preprocessed_data: tweak interface, add documentation

adammoody committed 4 years ago

Merge branch 'main' into mergescript

adammoody committed 4 years ago

merge: improvements after testing

adammoody committed 4 years ago

tests: serial and distributed merge

adammoody committed 4 years ago

avoid setting pythonpath within script

adammoody committed 4 years ago

merge script: fix typo in usage comments

adammoody committed 4 years ago

print default backend when not set in distributed merge

adammoody committed 4 years ago

Megatron-DeepSpeed update merge_preprocessed_data to use distributed merge #82 Merged