DeepSpeed
Refactor MoE and Groups API to simplify model creation and mangement
#1798

Merged

Refactor MoE and Groups API to simplify model creation and mangement #1798

jeffra merged 38 commits into master from amawa/moe-api-updates

add docs/proposal.

6f212850

save pending code.

9ce86f5a

add more changes.

bb008c9a

Add changes to groups.

08e8a455

use proper world group instead of None.

e431ef37

fix the world group by cloning it

d2b7a445

fix group creation and return.

ec484adb

fix typo

8cc33c08

another stupid typo.

966c36be

fix failed unit tests.

95796951

use the correct method to get num_experts

b222a7b3

cleanup the layer code.

39197e88

remove the old code

90e0c967

Move group creation to enginer. Fix several small things. Use helper …

3c9d379f

fix

55b897e0

Merge branch 'master' into amawa/moe-api-updates

21a08ba5

Zheweiyao/moe post release residual moe (#289) (#290)

bd74c0b3

feedback from Jeff/ammar review

292a33e2

Fix unit tests and add checks to create groups.

e149ea69

fix failing moe tests.

b39145a7

fix name typo

08b31911

pass ep_size down to the layer.

addb7920

return rank

d318b032

All test pass. Fix pr-moe test

b93f5628

add missing import

881cfc47

parallelism spec based group creation.

b9f2aefb

fix missing underscores.

3a1ea689

Fix module search.

5e9067d3

remove print.

477a415f

add missing _

61a2e5d3

fix typos, fix split parameters for moe optimizer

e5260071

update moe/cifar-moe tutorial.

40ce1b5b

explain/fix use_residual.

2713e0d3

fix unit tests and remove groups from tutorial. mark todo for inferen…

2e89e0fe

get the moe flag from the right place on the transformer layer

1fedae9b

Add in more feedback.

695945c3

Add in feedback

5323f746

Merge branch 'master' into amawa/moe-api-updates

7d5ae4bd