Fix DataLoader sharding for deepspeed in accelerate #315
sgugger
merged
2 commits into
huggingface:main
from
m3rlin45:fix_deepspeed_data_sharding
set first_pass on calls from deepspeed to _prepare_one(...) so that iā¦
68093042
sgugger
approved these changes
on 2022-04-13
sgugger3 years ago (edited 3 years ago)š 1
Very nice catch, thanks a lot for fixing!
Can you just run make style on your branch to fix the formatting issue?
HuggingFaceDocBuilderDev3 years ago (edited 3 years ago)
The documentation is not available anymore as the PR was closed or merged.
fixed style
e293f7d3
Login to write a write a comment.
Summary
I noticed that my training using the deepspeed integration in accelerate had some strange behavior, the number of iterations per epoch didn't go down as I increased number of GPUs. Also loss wasn't converging as expected.
After a bunch of debugging, I found that
_prepare_deepspeed(...)doesn't appear to call_prepare_one(...)properly. It calls without settingfirst_pass=True, which means thatprepare_one(...)skips wrapping the DataLoaders... defeating the whole pointHow I tested
I added logging to my training flow to print out
len(data_loader)afteraccelerator.prepare(...)is called.I validated that with this fix, the length is divided by the number of processes, as expected.