PR #5008 Delay reduce-scatter for ZeRO3 leaf modules

Delay reduce-scatter for ZeRO3 leaf modules #5008

mrwyattii merged 6 commits into master from tohtana/z3_moe_bwd

delay reduce-scatter for z3 leaf modules

93268f30

Merge branch 'master' into tohtana/z3_moe_bwd

a56ffec0

tohtana marked this pull request as ready for review 1 year ago

tohtana requested a review from

tjruwase 1 year ago

tohtana requested a review from

mrwyattii 1 year ago

tohtana requested a review from

awan-10 1 year ago

tohtana requested a review from

loadams 1 year ago

add api to get leaf modules

ca5cff6a

Merge branch 'master' into tohtana/z3_moe_bwd

08770b8c

tjruwase removed review request from

awan-10 1 year ago

tjruwase removed review request from

mrwyattii 1 year ago

tjruwase removed review request from

loadams 1 year ago

tjruwase requested a review from

GuanhuaWang 1 year ago

tjruwase removed review request from

GuanhuaWang 1 year ago

tjruwase approved these changes on 2024-02-01

Merge branch 'master' into tohtana/z3_moe_bwd

fccbd950

tohtana enabled auto-merge 1 year ago

Merge branch 'master' into tohtana/z3_moe_bwd

da1f41d3

mrwyattii merged 19e0dc39 into master 1 year ago

mrwyattii deleted the tohtana/z3_moe_bwd branch 1 year ago

Reviewers

tjruwase

Assignees

No one assigned

Labels

None yet

Milestone

No milestone