DeepSpeed
sequence parallel with communication overlap
#5691
Merged
Overview
Commits
20
Changes
View On GitHub

sequence parallel with communication overlap #5691

loadams merged 20 commits into deepspeedai:master from inkcherry:sp_overlap_comm
inkcherry
inkcherry fix ds-sp grad scale for zero0
cb15ffa1
inkcherry enable o compute async
a037a53f
inkcherry enable qk bwd async all2all
42d12849
inkcherry fwd optimi
6919af43
inkcherry fix1 remove linear arg, remove note
39596ac4
inkcherry async qkv fwd, optimi cpu ,make fwd call fast
eb760c01
inkcherry update
c7d3374a
inkcherry refine code
70a6d0c9
inkcherry refine code
65afd895
inkcherry Revert "fix ds-sp grad scale for zero0"
4b3518ed
inkcherry Merge remote-tracking branch 'upstream/master' into sp_overlap_comm
634d6d93
inkcherry fix format
54b5ce3d
inkcherry inkcherry requested a review from mrwyattii mrwyattii 1 year ago
inkcherry fix format
c9f0c0ad
tjruwase tjruwase removed review request from mrwyattii mrwyattii 1 year ago
tjruwase tjruwase requested a review from samadejacobs samadejacobs 1 year ago
tjruwase tjruwase requested a review from tohtana tohtana 1 year ago
Edenzzzz
inkcherry refine code
0862aa37
inkcherry add register for v, ensuring they launch on a single thread.
1c596dd6
inkcherry
tjruwase Merge branch 'master' into sp_overlap_comm
96e76962
inkcherry remove v
2fbbd5eb
inkcherry remove v
765a664f
inkcherry
inkcherry fix notes and format
171eb67e
HeyangQin Merge branch 'master' into sp_overlap_comm
020ab5f8
samadejacobs
tohtana
tohtana approved these changes on 2024-07-22
loadams loadams merged 17ed7c77 into master 1 year ago
Edenzzzz
inkcherry

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum