transformers
Enable FSDP2 + expert parallelism via a 2-D (fsdp, tp) device mesh
#48516
Merged
Overview
Commits
26
Changes
View On GitHub

Enable FSDP2 + expert parallelism via a 2-D (fsdp, tp) device mesh #48516

qgallouedec merged 26 commits into main from fsdp2-ep-2d-mesh
qgallouedec
HuggingFaceDocBuilderDev
qgallouedec
qgallouedec qgallouedec force pushed from 393af534 to d226b020 14 days ago
qgallouedec qgallouedec force pushed from d226b020 to 680e1ff1 11 days ago
qgallouedec qgallouedec force pushed from 680e1ff1 to d9394c9e 11 days ago
qgallouedec Fix NaN gradients in expert-parallel training: mask uninitialized gro…
8af1a002
qgallouedec Fix wrong gradients for all non-expert parameters in expert-parallel …
1f0b7bf6
qgallouedec Trim comments
4636d425
qgallouedec Gate the router-score backward all-reduce on grad mode; drop the post…
89fa55ca
qgallouedec Fix expert parallelism through Trainer (#48208)
96769d99
qgallouedec Enable FSDP2 + expert parallelism via a 2-D (fsdp, tp) device mesh
c74c1228
qgallouedec Gather the sharded state dict through the model's own save helpers
1c66acbc
qgallouedec Document combining expert parallelism with FSDP2
38aefbf2
qgallouedec Address review: drop the dead expert-gradient sync, torch-native mixe…
e72b117d
qgallouedec Give each device mesh its own optimizer param group instead of steppi…
449acd21
qgallouedec Test the 2-D mesh with a batch split across the fsdp ranks and compar…
9d17bd7a
qgallouedec Refresh the throughput table
6f2d9c4b
qgallouedec Cosmetic: docstring layout, comment placement, doc link
3a8da119
qgallouedec qgallouedec force pushed from d9394c9e to 3a8da119 10 days ago
Base automatically changed from fix-ep-training-gradients to main 10 days ago
qgallouedec Merge remote-tracking branch 'upstream/main' into fsdp2-ep-2d-mesh
0e21b598
stevhliu
stevhliu approved these changes on 2026-09-08
qgallouedec Merge branch 'main' into fsdp2-ep-2d-mesh
35ed430b
qgallouedec Update docs/source/en/expert_parallelism.md
75fff716
qgallouedec Update docs/source/en/expert_parallelism.md
acba6bc9
qgallouedec Update docs/source/en/expert_parallelism.md
69668c3c
qgallouedec Update docs/source/en/expert_parallelism.md
9cfb2d3b
qgallouedec Merge branch 'main' into fsdp2-ep-2d-mesh
2884e2f1
qgallouedec Merge branch 'main' into fsdp2-ep-2d-mesh
1be3a7f5
qgallouedec Merge branch 'main' into fsdp2-ep-2d-mesh
a6f5824a
3outeille
3outeille commented on 2026-09-11
3outeille
3outeille commented on 2026-09-11
qgallouedec Merge branch 'main' into fsdp2-ep-2d-mesh
cb02b27f
qgallouedec Merge remote-tracking branch 'upstream/main' into fsdp2-ep-2d-mesh
79988c35
qgallouedec Drop the comment on the parallel sizes
51207488
qgallouedec Trim the parallelism config comment
0147840f
3outeille
3outeille
3outeille approved these changes on 2026-09-12
qgallouedec qgallouedec enabled auto-merge 6 days ago
github-actions
qgallouedec qgallouedec merged 415e6d2f into main 6 days ago
qgallouedec qgallouedec deleted the fsdp2-ep-2d-mesh branch 6 days ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum