transformers
b56e1d68 - Merge branch 'fsdp2-ep-2d-mesh' into ep-token-dispatch

Commit

18 days ago

Merge branch 'fsdp2-ep-2d-mesh' into ep-token-dispatch

References

#48204 - [Umbrella] FSDP2 + expert-parallel token dispatch, with sharded-loading prefetch

#48518 - Expert-parallel token dispatch: each rank trains on its own part of the batch

#48792 - Add Expert parallel

Author

qgallouedec

qgallouedec

Parents

Loading