DeepSpeed
Write multiple items to output file at once, in distributed data analyzer.
#5169
Merged
Merged
Write multiple items to output file at once, in distributed data analyzer. #5169
conglongli
merged
45 commits into
deepspeedai:master
from
write_multiple_items_at_once_in_distributed_data_analyzer
added assert of torch vs numpy types
14f2bbe1
first draft
796341d9
reverted to original master
07aa4b42
added metric type accumulate_value_over_samples
815a7897
pre-commit
28a72e7c
Merge branch 'master' into distributed_data_analyzer
e8dbf0b3
Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepS…
ec3479fa
Update data_analyzer.py
38d7ce66
added check for single node reduce. added barriers
295fba67
more bug fixes
4144e427
new iteration, many bug fixes
a1e121c9
bug fixes
e045753c
Merge branch 'master' into distributed_data_analyzer
3a891162
fixing previous commit
cdc838c1
Merge branch 'master' into distributed_data_analyzer
ba34a550
pre-commit
5c077104
Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepS…
87d76867
write sequentially to file
a634787f
Merge branch 'master' into distributed_data_analyzer
848ffd5d
fixes in sequential write
ec59f08d
Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepS…
832874c2
pre-commit hooks
ea0d65f5
Merge branch 'master' into distributed_data_analyzer
c6c9bc5b
added main as example
56a95338
Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepS…
b4d86543
Merge branch 'master' into distributed_data_analyzer
676dc1a3
Update data_analyzer.py
6788af55
first working version. idx files differ
bd61d9c2
Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepS…
7ac5e45c
added missing static function
8bf0e635
removed/added breaklines to match base code
e5a7eb0f
corrected comment
3b8014fd
imports
5a426879
removed main
cdaad362
reverted main
b3d40620
bug fix in sample calculation
7cabfa2a
added worker_an and num_worker to kwargs
62f68dd1
removed dist.initialize ()from DataAnalyzer.run_map_reduce
6d35e454
first iteration
be91d37c
updated with add_items
5fd05468
added add_items
f5be5e1b
Merge branch 'master' into write_multiple_items_at_once_in_distribute…
1ccd3bad
Update indexed_dataset.py
9d5c1715
bm-synth
changed the title
bm-synth
changed the title 
Merge branch 'master' into write_multiple_items_at_once_in_distribute…
dc1dbb30
conglongli
approved these changes
on 2024-02-21
formatting
db29942f
Login to write a write a comment.
Login via GitHub