DeepSpeed
Write multiple items to output file at once, in distributed data analyzer.
#5169
Merged
Overview
Commits
45
Changes
View On GitHub
Commits
  • added assert of torch vs numpy types
    bm-synth committed 2 years ago
  • first draft
    bm-synth committed 2 years ago
  • reverted to original master
    bm-synth committed 2 years ago
  • added metric type accumulate_value_over_samples
    bm-synth committed 2 years ago
  • pre-commit
    bm-synth committed 2 years ago
  • Merge branch 'master' into distributed_data_analyzer
    bm-synth committed 2 years ago
  • Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepSpeed into distributed_data_analyzer
    bm-synth committed 2 years ago
  • Update data_analyzer.py
    bm-synth committed 2 years ago
  • added check for single node reduce. added barriers
    bm-synth committed 2 years ago
  • more bug fixes
    bm-synth committed 2 years ago
  • new iteration, many bug fixes
    bm-synth committed 2 years ago
  • bug fixes
    bm-synth committed 2 years ago
  • Merge branch 'master' into distributed_data_analyzer
    bm-synth committed 2 years ago
  • fixing previous commit
    bm-synth committed 2 years ago
  • Merge branch 'master' into distributed_data_analyzer
    bm-synth committed 2 years ago
  • pre-commit
    bm-synth committed 2 years ago
  • Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepSpeed into distributed_data_analyzer
    bm-synth committed 2 years ago
  • write sequentially to file
    bm-synth committed 2 years ago
  • Merge branch 'master' into distributed_data_analyzer
    bm-synth committed 2 years ago
  • fixes in sequential write
    bm-synth committed 2 years ago
  • Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepSpeed into distributed_data_analyzer
    bm-synth committed 2 years ago
  • pre-commit hooks
    bm-synth committed 2 years ago
  • Merge branch 'master' into distributed_data_analyzer
    bm-synth committed 2 years ago
  • added main as example
    bm-synth committed 2 years ago
  • Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepSpeed into distributed_data_analyzer
    bm-synth committed 2 years ago
  • Merge branch 'master' into distributed_data_analyzer
    bm-synth committed 2 years ago
  • Update data_analyzer.py
    bm-synth committed 2 years ago
  • first working version. idx files differ
    bm-synth committed 2 years ago
  • Merge branch 'distributed_data_analyzer' of github.com:bm-synth/DeepSpeed into distributed_data_analyzer
    bm-synth committed 2 years ago
  • added missing static function
    bm-synth committed 2 years ago
  • removed/added breaklines to match base code
    bm-synth committed 2 years ago
  • corrected comment
    bm-synth committed 2 years ago
  • imports
    bm-synth committed 2 years ago
  • removed main
    bm-synth committed 2 years ago
  • reverted main
    bm-synth committed 2 years ago
  • bug fix in sample calculation
    bm-synth committed 2 years ago
  • added worker_an and num_worker to kwargs
    bm-synth committed 2 years ago
  • removed dist.initialize ()from DataAnalyzer.run_map_reduce
    bm-synth committed 2 years ago
  • first iteration
    bm-synth committed 2 years ago
  • updated with add_items
    bm-synth committed 2 years ago
  • added add_items
    bm-synth committed 2 years ago
  • Merge branch 'master' into write_multiple_items_at_once_in_distributed_data_analyzer
    bm-synth committed 2 years ago
  • Update indexed_dataset.py
    bm-synth committed 2 years ago
  • Merge branch 'master' into write_multiple_items_at_once_in_distributed_data_analyzer
    conglongli committed 2 years ago
  • formatting
    bm-synth committed 2 years ago
FAQ Terms Privacy Refunds Impressum
Loading