PR #1188 ZeRO3, improved parameter all-gather operation

ZeRO3, improved parameter all-gather operation #1188

tjruwase merged 39 commits into deepspeedai:master from zarzen:impr_allgather_params

remove norm(), avoid memcpy after allgather

1e73e758

zarzen requested a review from

arashashari 4 years ago

zarzen requested a review from

awan-10 4 years ago

zarzen requested a review from

cli99 4 years ago

zarzen requested a review from

conglongli 4 years ago

zarzen requested a review from

eltonzheng 4 years ago

zarzen requested a review from

jeffra 4 years ago

zarzen requested a review from

minjiaz 4 years ago

zarzen requested a review from

niumanar 4 years ago

zarzen requested a review from

RezaYazdaniAminabadi 4 years ago

zarzen requested a review from

samyam 4 years ago

zarzen requested a review from

ShadenSmith 4 years ago

zarzen requested a review from

tjruwase 4 years ago

zarzen marked this pull request as draft 4 years ago

zarzen force pushed to 1e73e758 4 years ago

WIP: wrapped ncclAllgather as customized op in DS

67b3db3e

WIP: integrated into partition_parameters

70e681f0

Fix format

81b4fc4a

zarzen marked this pull request as ready for review 4 years ago

Merge branch 'master' into impr_allgather_params

8a14e434

cleaned dead code, modified unit test

32c8fa72

Merge branch 'master' into impr_allgather_params

c4728f50

Merge branch 'master' into impr_allgather_params

e075fd4d

tjruwase removed review request from

conglongli 4 years ago

tjruwase removed review request from

awan-10 4 years ago

tjruwase removed review request from

arashashari 4 years ago

tjruwase removed review request from

cli99 4 years ago

tjruwase removed review request from

eltonzheng 4 years ago

tjruwase removed review request from

minjiaz 4 years ago

tjruwase removed review request from

RezaYazdaniAminabadi 4 years ago

tjruwase removed review request from

niumanar 4 years ago

removed customized c++ extension

52085080

Merge remote-tracking branch 'origin/master' into impr_allgather_params

ffd3d3b0

change torch.ones to torch empty

1ed96ce2

Merge branch 'master' into impr_allgather_params

220f2e0e

Merge branch 'master' into impr_allgather_params

8f655940

typo

0e6d8e0c

Merge branch 'master' into impr_allgather_params

691749fe

Merge branch 'master' into impr_allgather_params

88e750e6

Merge branch 'master' into impr_allgather_params

497ee7d7

Merge branch 'master' into impr_allgather_params

bd8839c5

Merge branch 'master' into impr_allgather_params

25829106

Merge branch 'master' into impr_allgather_params

4ca0d391

Merge branch 'master' into impr_allgather_params

56de9ad9

Merge branch 'master' into impr_allgather_params

056cf101

Merge branch 'master' into impr_allgather_params

aac09cd2

jeffra commented on 2021-10-07

Merge branch 'master' into impr_allgather_params

6201b295

warn if not cuda tensor for allgather

50a9215d

Merge branch 'master' into impr_allgather_params

c554a589

Merge branch 'master' into impr_allgather_params

b7e131d4

fix formatting

813cb227

Merge branch 'master' into impr_allgather_params

588d3d0e

Merge branch 'master' into impr_allgather_params

eb0a540f

fix: move ds_tensor to cuda device

c092b789

Merge branch 'master' into impr_allgather_params

e73809dc

Merge branch 'master' into impr_allgather_params

d1d3c28f

tjruwase commented on 2021-10-27

Merge branch 'master' into impr_allgather_params

62cb1049

Merge branch 'master' into impr_allgather_params

ab64b17a

remove try clause on the path for fetching params

7a801721

Merge branch 'microsoft:master' into impr_allgather_params

f01dad85

tjruwase approved these changes on 2021-10-30

Merge branch 'master' into impr_allgather_params

524e6096

Merge branch 'master' into impr_allgather_params

d7fff585

tjruwase enabled auto-merge (squash) 4 years ago

tjruwase merged c0eeb69d into master 4 years ago

Reviewers

tjruwase

jeffra

samyam

ShadenSmith

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

DeepSpeed ZeRO3, improved parameter all-gather operation #1188 Merged

ZeRO3, improved parameter all-gather operation #1188

DeepSpeed
ZeRO3, improved parameter all-gather operation
#1188

Merged