DeepSpeed
Fix potential memory issues when use deepspeed Z3
#6726

Merged

Fix potential memory issues when use deepspeed Z3 #6726

loadams merged 9 commits into deepspeedai:master from wenbinc-Bin:fix_z3

wenbinc-Bin requested a review from

tjruwase 1 year ago

tjruwase commented on 2024-11-12

tjruwase requested a review from

tohtana 1 year ago

wenbinc-Bin force pushed from 12189feb to 93ef46ad 1 year ago

wenbinc-Bin requested a review from

loadams 1 year ago

wenbinc-Bin force pushed from 93ef46ad to 7cd4f335 1 year ago

Set "ds_grads_remaining" to 0 when module doesn't have this variable

d75d1f5c

Set "__n_available_params" to 0 in release_and_reset_all()

b83df91c

Add unit stage3 test for running model twice in one step

9f0d2aee

wenbinc-Bin force pushed from 7cd4f335 to 9f0d2aee 1 year ago

tjruwase approved these changes on 2024-11-18

Merge branch 'master' into fix_z3

ad60f943

Merge branch 'master' into fix_z3

8b30987d

Merge branch 'master' into fix_z3

7834887a

Merge branch 'master' into fix_z3

51a69c07

loadams enabled auto-merge 1 year ago

Fix dtype error

7cabf80a

disabled auto-merge 1 year ago
Head branch was pushed to by a user without write access

Merge branch 'master' into fix_z3

ed40ac77

loadams merged cd20a3bb into master 1 year ago

Reviewers

tjruwase

tohtana

loadams

Assignees

No one assigned

Labels

None yet

Milestone

No milestone