transformers
[`distributed`] Context parallelism with expert parallelism
#50514

Open

[`distributed`] Context parallelism with expert parallelism #50514

3outeille wants to merge 2 commits into cp-6-tp-x-cp from cp-9-ep-x-cp

3outeille force-pushed the cp-6-tp-x-cp branch from 5acd8bdb to 2b71a64e 1 day ago

3outeille force pushed from c9e600a7 to d0d2eb7e 1 day ago

3outeille force-pushed the cp-6-tp-x-cp branch from 2b71a64e to 210c54b1 1 day ago

3outeille force pushed from d0d2eb7e to 58051e90 1 day ago

3outeille force-pushed the cp-6-tp-x-cp branch from 210c54b1 to 59258f74 15 hours ago

3outeille force pushed from 58051e90 to 62333ac7 15 hours ago

3outeille force-pushed the cp-6-tp-x-cp branch from 59258f74 to 37cf5842 15 hours ago

3outeille force pushed from 62333ac7 to 963136f9 15 hours ago

3outeille force-pushed the cp-6-tp-x-cp branch from 37cf5842 to a64ce5d6 12 hours ago

3outeille force pushed from 963136f9 to dcd89f0d 12 hours ago

3outeille force-pushed the cp-6-tp-x-cp branch from a64ce5d6 to 9a8e25a0 12 hours ago

3outeille force pushed from dcd89f0d to 95dc69d9 12 hours ago

[distributed] Context parallelism with expert parallelism

4c602b3e

[distributed] Test expert parallelism with context parallelism

55ebcac0

3outeille force-pushed the cp-6-tp-x-cp branch from 9a8e25a0 to 752288db 12 hours ago

3outeille force pushed from 95dc69d9 to 55ebcac0 12 hours ago

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone