DeepSpeed
Add sequential allgather optimization for ZeRO-3
#7661

Merged

Add sequential allgather optimization for ZeRO-3 #7661

sfc-gh-truwase merged 8 commits into deepspeedai:master from aeeeeeep:allgather_single_param

aeeeeeep requested a review from

tjruwase 203 days ago

aeeeeeep requested a review from

tohtana 203 days ago

aeeeeeep force pushed from f5805587 to 77a51f7e 203 days ago

aeeeeeep marked this pull request as draft 202 days ago

aeeeeeep force pushed from 3613b25d to d55f7363 202 days ago

aeeeeeep marked this pull request as ready for review 202 days ago

aeeeeeep force pushed from d55f7363 to 30814fa3 202 days ago

aeeeeeep force pushed from efecf04d to d6cd73de 202 days ago

aeeeeeep force pushed from 52ae9614 to c943e0e4 190 days ago

aeeeeeep force pushed from a2fcbb8d to 47f0febf 127 days ago

aeeeeeep force pushed from 47f0febf to c5da649c 127 days ago

sfc-gh-truwase commented on 2026-01-15

aeeeeeep changed the title ~~Add single parameter allgather optimization for zero3~~ Add sequential allgather optimization for ZeRO-3 124 days ago

Add single parameter allgather optimization for zero3

e0412070

adaptor func _allgather_params

b9dbf314

fix undefined name

fca9157b

fix prefetch bucket size

ac07ca16

format

4ecbbb79

update config comment

d3b8e814

fix dtype mismatch in quantize

413bc60c

refactor code

c37ae66b

aeeeeeep force pushed from 2f37304b to c37ae66b 123 days ago

sfc-gh-truwase approved these changes on 2026-01-19

sfc-gh-truwase merged 60876090 into master 123 days ago

aeeeeeep deleted the allgather_single_param branch 119 days ago

Reviewers

sfc-gh-truwase

tjruwase

tohtana

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

DeepSpeed Add sequential allgather optimization for ZeRO-3 #7661 Merged

Add sequential allgather optimization for ZeRO-3 #7661

DeepSpeed
Add sequential allgather optimization for ZeRO-3
#7661

Merged