Megatron-DeepSpeed
update merge_preprocessed_data to use distributed merge
#82
Merged
Merged
update merge_preprocessed_data to use distributed merge #82
thomasw21
merged
9 commits into
bigscience-workshop:main
from
adammoody:mergescript
update merge_preprocessed_data to use parallel merge
cb5e674b
adammoody
changed the title
indexed_dataset: add docstrings to merge and gather methods
ab81641b
merge_preprocessed_data: tweak interface, add documentation
ff37f443
Merge branch 'main' into mergescript
9babb815
merge: improvements after testing
cde15ea6
tests: serial and distributed merge
cdb74778
avoid setting pythonpath within script
549b3fe9
merge script: fix typo in usage comments
942fcdad
print default backend when not set in distributed merge
ff63c320
adammoody
changed the title
Login to write a write a comment.
Login via GitHub