Parallel map step for `DistributedDataAnalyzer` map-reduce #5291
conglongli
merged
60 commits into
deepspeedai:master
from
parallel_run_in_distributed_data_analyzer
added assert of torch vs numpy types
14f2bbe1
first draft
796341d9
reverted to original master
07aa4b42
added metric type accumulate_value_over_samples
815a7897
pre-commit
28a72e7c
Merge branch 'master' into distributed_data_analyzer
e8dbf0b3
Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepS…
ec3479fa
Update data_analyzer.py
38d7ce66
added check for single node reduce. added barriers
295fba67
more bug fixes
4144e427
new iteration, many bug fixes
a1e121c9
bug fixes
e045753c
Merge branch 'master' into distributed_data_analyzer
3a891162
fixing previous commit
cdc838c1
Merge branch 'master' into distributed_data_analyzer
ba34a550
pre-commit
5c077104
Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepS…
87d76867
recoverd master branch
f28e829b
write sequentially to file
a634787f
Merge branch 'master' into distributed_data_analyzer
848ffd5d
fixes in sequential write
ec59f08d
Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepS…
832874c2
pre-commit hooks
ea0d65f5
Merge branch 'master' into distributed_data_analyzer
c6c9bc5b
added main as example
56a95338
Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepS…
b4d86543
Merge branch 'master' into distributed_data_analyzer
676dc1a3
Update data_analyzer.py
6788af55
first working version. idx files differ
bd61d9c2
Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepS…
7ac5e45c
added missing static function
8bf0e635
removed/added breaklines to match base code
e5a7eb0f
corrected comment
3b8014fd
imports
5a426879
removed main
cdaad362
reverted main
b3d40620
bug fix in sample calculation
7cabfa2a
added worker_an and num_worker to kwargs
62f68dd1
removed dist.initialize ()from DataAnalyzer.run_map_reduce
6d35e454
first iteration
be91d37c
updated with add_items
5fd05468
Merge branch 'master' into serial_data_analyzer
e943aaa7
master in line with remote
f6c5c18d
merge with master
2e2ebee4
first iteration, needs testing and debugging on num_threads>1
e4de5545
first draft of multiprocessing
dadf0f92
first draft of multiprocessing
0ba9f930
pre-commit hooks
a761196a
bm-synth
changed the title
Merge branch 'master' into parallel_run_in_distributed_data_analyzer
7b9d1c66
pre-commit hooks and merge conflicts
c5dad209
faster writes
4313e8df
pre-commit hooks
34965a04
Merge branch 'master' into parallel_run_in_distributed_data_analyzer
fb5f8fa6
Merge branch 'master' into parallel_run_in_distributed_data_analyzer
2c30081a
Merge branch 'master' into parallel_run_in_distributed_data_analyzer
6d26d1aa
Merge branch 'master' into parallel_run_in_distributed_data_analyzer
afa7495d
Merge branch 'master' into parallel_run_in_distributed_data_analyzer
6106757e
Merge branch 'master' into parallel_run_in_distributed_data_analyzer
6abfdda5
Merge branch 'master' into parallel_run_in_distributed_data_analyzer
444644b2
Merge branch 'master' into parallel_run_in_distributed_data_analyzer
b130322f
conglongli
approved these changes
on 2024-04-18
Login to write a write a comment.
Login via GitHub