Enable torch.compile with ZeRO (Experimental) #4878
tohtana
merged
49 commits into
master
from
tohtana/compile-zero
add option to run torch.compile
91c01ab6
improve compile helper
49c7acd1
fix compile wrapper to make modules() work
bbeb38cf
add torch.compiler-disable to comm module
719cc79d
move options for torch.compile to ds config
3ea9b444
Merge branch 'master' into tohtana/compile-zero
eb9d4e06
rename module and wrap decorator
fbae0278
fix validation of compile config
4f8f86d6
avoid reference to torch._dynamo when torch has no support
d83963bd
fix custom backend for test
6920ab60
fix validation
c3429a68
refactor config for torch.compile
bfafb886
Merge branch 'master' into tohtana/compile-zero
3c13fd41
Merge branch 'master' into tohtana/compile-zero
9e63c956
fix validation of compiler config
ff9c1efb
fix access to wrapped model
26b7f254
add test for api to set torch compile options
48d2453f
rename util module
d5584b00
fix import
b9157ac5
Merge branch 'master' into tohtana/compile-zero
c19bf97a
delay reduce-scatter for z3 leaf modules
93268f30
Merge branch 'master' into tohtana/z3_moe_bwd
a56ffec0
add comment to config class
2a5e741d
Merge branch 'master' into tohtana/compile-zero
ec919251
add api to get leaf modules
ca5cff6a
Merge branch 'master' into tohtana/compile-zero
f6151380
Merge branch 'master' into tohtana/z3_moe_bwd
08770b8c
add api to set a function to run torch.compile
3e5658be
Merge branch 'master' into tohtana/compile-zero
5d9992e2
tjruwase
commented
on 2024-01-31
tjruwase
commented
on 2024-01-31
tjruwase
commented
on 2024-01-31
tjruwase
commented
on 2024-01-31
tjruwase
commented
on 2024-01-31
refactor compile config
a3c0e5d3
lift is_compile_supported up to use as `deepspeed.is_compile_supporte…
95f4f344
avoid overwriting backend fn in validator
1932b787
add tests combining compile and zero
ca856052
rename test modules
19dd4548
Merge branch 'master' into tohtana/z3_moe_bwd
fccbd950
Merge branch 'master' into tohtana/z3_moe_bwd
da1f41d3
use no zero + no compile as baseline for tests
d8c0a14c
disable memory_efficient_linear when torch.compile is enabled
ca419b4d
pass only tensors to z3 hooks to prevent dynamo from displaying errors
5e3a0707
Merge branch 'tohtana/z3_moe_bwd' into tohtana/compile-zero
06379688
Merge branch 'master' into tohtana/compile-zero
7131d6e5
fix exception used in test
96c8647f
increse tolerance in tests
0b3dae9b
add check for bf16
94cc97a5
enable accelerator check for bf16
eb27b9d3
Merge branch 'master' into tohtana/compile-zero
c1e10e5e
tjruwase
approved these changes
on 2024-02-05
update DistributedTest to work with torch.compile tests
c2ba829d
remove unused global
82f80d12
mrwyattii
approved these changes
on 2024-02-05
Merge branch 'master' into tohtana/compile-zero
cb540d35
oraluben
commented
on 2024-02-18
Login to write a write a comment.
Login via GitHub