Megatron-DeepSpeed
Faster preprocessing
#18
Merged
Overview
Commits
15
Changes
View On GitHub
Commits
  • Propose a faster preprocessing mechanim by reducing the interprocesses communications
    Thomas committed 4 years ago
  • Add flush in order to force print
    Thomas committed 4 years ago
  • Try to prevent dead locks
    Thomas committed 4 years ago
  • Woops
    Thomas committed 4 years ago
  • Trying to figure out what causes deadlock
    Thomas committed 4 years ago
  • Limit queue size to 1_000_000
    Thomas committed 4 years ago
  • Drastically reduce the maximum number of element in the queue
    Thomas committed 4 years ago
  • Threading does not use a worker
    thomasw21 committed 4 years ago
  • Remove shard files and factorise shard naming
    thomasw21 committed 4 years ago
  • Document high number of worker preprocessing script
    thomasw21 committed 4 years ago
  • Improve naming
    thomasw21 committed 4 years ago
  • Update comments and readmes
    thomasw21 committed 4 years ago
  • Woops
    thomasw21 committed 4 years ago
  • Remove the notion of vanilla and point to the script instead
    thomasw21 committed 4 years ago
  • Rephrase readme to use around 60 cores instead of 40
    thomasw21 committed 4 years ago
FAQ Terms Privacy Refunds Impressum
Loading