PR #716 Run HF dataset processing on local rank 0 first

Run HF dataset processing on local rank 0 first #716

dakinggg merged 17 commits into mosaicml:main from dakinggg:less-procs

less aggressive

49e6ce19

dakinggg requested a review from

irenedea 2 years ago

dakinggg marked this pull request as ready for review 2 years ago

missed one

14aef60f

precommit

e183cd62

add desc

79c20d01

fix

1b29b127

dakinggg marked this pull request as draft 2 years ago

attempt

329bc50d

clean up

90ceb6c5

add logs

f5cfd9f2

less procs

7a2867aa

move the barrier

55a4e846

precommit

95f4cf58

dakinggg marked this pull request as ready for review 2 years ago

dakinggg requested a review from

j316chuck 2 years ago

dakinggg requested a review from

mvpatel2000 2 years ago

dakinggg requested a review from

XiaohanZhangCMU 2 years ago

try again

d0af2b0c

precommit

72872d1e

xiaohanzhan-db approved these changes on 2023-11-06

karan6181 commented on 2023-11-06

xiaohanzhan-db requested changes on 2023-11-06

dakinggg changed the title ~~Less aggressive multiprocessing for map/filter~~ Run multiprocessed HF dataset processing on local rank 0 first 2 years ago

dakinggg changed the title ~~Run multiprocessed HF dataset processing on local rank 0 first~~ Improved map/filter for HF dataset processing 2 years ago

Merge branch 'main' into less-procs

a314e902

irenedea approved these changes on 2023-11-06

dakinggg changed the title ~~Improved map/filter for HF dataset processing~~ Run HF dataset processing on local rank 0 first 2 years ago

mvpatel2000 commented on 2023-11-06

move barrier around

397ba531

Merge branch 'main' into less-procs

c36cbe20

comments

e9ce00f9

mvpatel2000 approved these changes on 2023-11-06

dakinggg merged c2f5742d into main 2 years ago

dakinggg deleted the less-procs branch 2 years ago

Reviewers

irenedea

mvpatel2000

xiaohanzhan-db

karan6181

j316chuck

XiaohanZhangCMU

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

llm-foundry Run HF dataset processing on local rank 0 first #716 Merged

Run HF dataset processing on local rank 0 first #716

llm-foundry
Run HF dataset processing on local rank 0 first
#716

Merged