DeepSpeed
Support MoE for pipeline models
#5338
Merged
Overview
Commits
11
Changes
View On GitHub

Support MoE for pipeline models #5338

loadams merged 11 commits into deepspeedai:master from mosheisland:moe/pipe
mosheisland
mosheisland mosheisland requested a review from tjruwase tjruwase 1 year ago
mosheisland mosheisland requested a review from mrwyattii mrwyattii 1 year ago
mosheisland mosheisland requested a review from duli2012 duli2012 1 year ago
mosheisland mosheisland requested a review from awan-10 awan-10 1 year ago
mosheisland mosheisland requested a review from arashb arashb 1 year ago
mosheisland mosheisland requested a review from loadams loadams 1 year ago
mosheisland
tohtana tohtana requested a review from tohtana tohtana 1 year ago
MOE: Support bf16 grads reduce for pipeline
0050fdad
MOE: Use backward compatible methods to access tp info
d04cb9cc
MOE: Enable save MoE checkpoint for Pipeline models
f5c4d1a4
MOE: Support display of MoE loss for Pipeline models
a0e80123
MOE: Fix loading checkpoint of Pipeline models
b20db806
MOE: Fix group for max capacity all-reduce
a46f35d5
MOE: Enhance expert group creation for pipeline
d8ecc22b
MOE: Update global norm calculation for pipeline
0f9d2b58
mosheisland mosheisland force pushed from 2efbf80a to 7a5e8881 1 year ago
mosheisland mosheisland force pushed from 7a5e8881 to 0f9d2b58 1 year ago
tohtana
tohtana approved these changes on 2024-04-04
tohtana
MOE: fix style issue in pipe load_module_state_dict
b6067d7e
mosheisland
mosheisland Merge branch 'master' into moe/pipe
526ce7f2
mosheisland Merge branch 'master' into moe/pipe
4d8bf271
tohtana tohtana enabled auto-merge 1 year ago
loadams loadams merged 08e0733e into master 1 year ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum