PR #5220 Fix moe cpu offload

Fix moe cpu offload #5220

mrwyattii merged 6 commits into deepspeedai:master from RezaYazdaniAminabadi:fix-moe-cpu-offload

Skip gradient-norm averaging when cpu-device is selected

feb578af

fix formatting

9eb6129e

RezaYazdaniAminabadi requested a review from

tjruwase 1 year ago

RezaYazdaniAminabadi requested a review from

mrwyattii 1 year ago

Merge branch 'master' into fix-moe-cpu-offload

9a59df40

tjruwase commented on 2024-03-02

average the grad-norms by sending the gradients to GPU when using off…

ae4bc936

Merge branch 'fix-moe-cpu-offload' of https://github.com/RezaYazdaniA…

d8255e07

Merge branch 'master' into fix-moe-cpu-offload

4fc1d8b9

tjruwase approved these changes on 2024-03-04

mrwyattii merged e6e8c137 into master 1 year ago

Reviewers

tjruwase

mrwyattii

Assignees

No one assigned

Labels

None yet

Milestone

No milestone