PR #5008 Delay reduce-scatter for ZeRO3 leaf modules

Delay reduce-scatter for ZeRO3 leaf modules #5008

mrwyattii merged 6 commits into master from tohtana/z3_moe_bwd

delay reduce-scatter for z3 leaf modules

93268f30

Merge branch 'master' into tohtana/z3_moe_bwd

a56ffec0

tohtana marked this pull request as ready for review 2 years ago

tohtana requested a review from

tjruwase 2 years ago

tohtana requested a review from

mrwyattii 2 years ago

tohtana requested a review from

awan-10 2 years ago

tohtana requested a review from

loadams 2 years ago

add api to get leaf modules

ca5cff6a

Merge branch 'master' into tohtana/z3_moe_bwd

08770b8c

tjruwase removed review request from

awan-10 2 years ago

tjruwase removed review request from

mrwyattii 2 years ago

tjruwase removed review request from

loadams 2 years ago

tjruwase requested a review from

GuanhuaWang 2 years ago

tjruwase removed review request from

GuanhuaWang 2 years ago

tjruwase approved these changes on 2024-02-01

Merge branch 'master' into tohtana/z3_moe_bwd

fccbd950

tohtana enabled auto-merge 2 years ago

Merge branch 'master' into tohtana/z3_moe_bwd

da1f41d3

mrwyattii merged 19e0dc39 into master 2 years ago

mrwyattii deleted the tohtana/z3_moe_bwd branch 2 years ago

Reviewers

tjruwase

Assignees

No one assigned

Labels

None yet

Milestone

No milestone