DeepSpeed
fix sequence parallel(Ulysses) grad scale for zero0
#5555

Merged

fix sequence parallel(Ulysses) grad scale for zero0 #5555

samadejacobs merged 2 commits into deepspeedai:master from inkcherry:fix_Ulysses_grad

fix ds-sp grad scale for zero0

cb15ffa1

inkcherry requested a review from

mrwyattii 1 year ago

inkcherry requested a review from

tjruwase 1 year ago

tjruwase removed review request from

tjruwase 1 year ago

tjruwase removed review request from

mrwyattii 1 year ago

tjruwase requested a review from

samadejacobs 1 year ago

tjruwase requested a review from

tohtana 1 year ago

samadejacobs commented on 2024-05-24

keep the correct dp_size

60e0dbc0

tohtana approved these changes on 2024-06-05

samadejacobs merged 6b6d6418 into master 1 year ago

Reviewers

tohtana

samadejacobs

Assignees

No one assigned

Labels

None yet

Milestone

No milestone