DeepSpeed
[Zero++ qgZ] Fall back to reduce_scatter if `tensor.numel() % (2 * global_world_size) != 0`
#5056

Merged

[Zero++ qgZ] Fall back to reduce_scatter if `tensor.numel() % (2 * global_world_size) != 0` #5056

loadams merged 4 commits into deepspeedai:master from ByronHsu:byhsu/a2a-bug

ByronHsu requested a review from

awan-10 2 years ago

ByronHsu requested a review from

mrwyattii 2 years ago

ByronHsu requested a review from

tjruwase 2 years ago

ByronHsu requested a review from

loadams 2 years ago

GuanhuaWang assigned

GuanhuaWang 2 years ago

GuanhuaWang commented on 2024-02-02

Fix all-to-all bug

cf5471ee

typo

50b37a92

format

647a0b17

take suggestions

0c5cb28e

ByronHsu force pushed from f2c7e65c to 0c5cb28e 2 years ago

GuanhuaWang approved these changes on 2024-02-03

loadams merged 592325ab into master 2 years ago

Reviewers

GuanhuaWang

awan-10

mrwyattii

tjruwase

loadams

Assignees

GuanhuaWang

Labels

None yet

Milestone

No milestone