DeepSpeed
encoded ds config into command line argument when launching child processes in autotuning
#2524
Merged
Merged
encoded ds config into command line argument when launching child processes in autotuning #2524
cli99
merged
20 commits into
master
from
encoded-ds-config
rollback ds config changes
d5e9a3cb
fix format
8f6517ad
Fix error when output_file is a relative path without a prefix (#2397)
b685aeb1
fix restuls and exprs path to use absolute path
f8c95138
use base64 encoded ds config as cmd arg
cd0f6eae
fix format
154890b1
remove assert
fb86d551
Merge branch 'master' into encoded-ds-config
a9198073
write out optimial config after tuning
20c1b1a1
fix format
d3f34920
no need to update ds config path when encoding ds config
0154d346
udpate
14080499
merge with master
27e1db45
do not use abs path for result and expr dir
1a9623e6
fix conflicts
c098e687
fix run mode
7df118e3
Merge branch 'master' into encoded-ds-config
73a29574
fix format
d6104115
fix format
b833a798
mrwyattii
approved these changes
on 2022-11-22
Merge branch 'master' into encoded-ds-config
35b65085
Login to write a write a comment.
This PR encodes the autotuning experiments ds configurations passed from the master tuning process to the child process. This is a required fix for AML environment, where child process cannot read a path wrote by the master process on a shared storage.
Huggingface assumes deepspeed_config is either a dictionary or a file path, to make it work with the PR for autotuning, the check needs to be updated as well.