Distributed in-memory map-reduce for data analyzer #5129
conglongli
merged
39 commits into
deepspeedai:master
from
distributed_data_analyzer
added assert of torch vs numpy types
14f2bbe1
first draft
796341d9
reverted to original master
07aa4b42
added metric type accumulate_value_over_samples
815a7897
pre-commit
28a72e7c
Merge branch 'master' into distributed_data_analyzer
e8dbf0b3
Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepS…
ec3479fa
Update data_analyzer.py
38d7ce66
added check for single node reduce. added barriers
295fba67
more bug fixes
4144e427
new iteration, many bug fixes
a1e121c9
bug fixes
e045753c
Merge branch 'master' into distributed_data_analyzer
3a891162
fixing previous commit
cdc838c1
Merge branch 'master' into distributed_data_analyzer
ba34a550
pre-commit
5c077104
Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepS…
87d76867
write sequentially to file
a634787f
Merge branch 'master' into distributed_data_analyzer
848ffd5d
fixes in sequential write
ec59f08d
Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepS…
832874c2
pre-commit hooks
ea0d65f5
Merge branch 'master' into distributed_data_analyzer
c6c9bc5b
bm-synth
changed the title
added main as example
56a95338
Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepS…
b4d86543
bm-synth
changed the title
bm-synth
changed the title
Merge branch 'master' into distributed_data_analyzer
676dc1a3
bm-synth
changed the title
Update data_analyzer.py
6788af55
first working version. idx files differ
bd61d9c2
Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepS…
7ac5e45c
conglongli
approved these changes
on 2024-02-19
added missing static function
8bf0e635
removed/added breaklines to match base code
e5a7eb0f
corrected comment
3b8014fd
imports
5a426879
removed main
cdaad362
reverted main
b3d40620
bug fix in sample calculation
7cabfa2a
added worker_an and num_worker to kwargs
62f68dd1
removed dist.initialize ()from DataAnalyzer.run_map_reduce
6d35e454
Merge branch 'master' into distributed_data_analyzer
c6292f23
Login to write a write a comment.
Login via GitHub