Fix NaN gradients in expert-parallel training: mask uninitialized gro…
8af1a002
Fix wrong gradients for all non-expert parameters in expert-parallel …
1f0b7bf6
Trim comments
4636d425
Gate the router-score backward all-reduce on grad mode; drop the post…
89fa55ca
Fix expert parallelism through Trainer (#48208)
96769d99
Enable FSDP2 + expert parallelism via a 2-D (fsdp, tp) device mesh
c74c1228
Gather the sharded state dict through the model's own save helpers
1c66acbc
Document combining expert parallelism with FSDP2
38aefbf2
Address review: drop the dead expert-gradient sync, torch-native mixe…
e72b117d
Give each device mesh its own optimizer param group instead of steppi…
449acd21
Test the 2-D mesh with a batch split across the fsdp ranks and compar…
9d17bd7a
Refresh the throughput table
6f2d9c4b
Cosmetic: docstring layout, comment placement, doc link
3a8da119
Expert-parallel token dispatch: each rank trains on its own part of t…
1b071993
Make the dispatch handling of the experts explicit in apply_fully_sha…
7063c456
Combine the dispatched expert outputs in the routing weights' precisi…
dc1fdeee
Split the batches across every rank with a DistributedSampler under t…
f2a3c26d
Simplify the dispatch: torch's differentiable all-to-all, the grouped…
3ab94ff1
Dispatch as plan-level styles; one notion of the ranks sharing a batc…
d1881444
Select the dispatch styles in the mixin, guard the torch version and …
b9bf58bd
Rewrite the expert parallel plan the model actually reads for the dis…
39cdce50
Seed the dispatch sampler with data_seed; require the grouped_mm expe…
9f862308
Run the local experts directly: the grouped forward's masking and com…
930c146a
Run the local experts through the module's own forward
15588e00
Dispatch: one host sync per layer, comments say what the code does
51f281d9
Dispatch: drop the router style, reuse the named mesh and is_dtensor,…
101bfea9
Dispatch: skip the mixin tests for models without an FSDP plan, drop …
2d9046bd
Fix the _has_fsdp_plan docstring
42606afb
Drop the FSDP plan skip from the mixin tests: every config carries a …
d8eb1fc1
Format
23920e3a
Keep the base install_forward signature on the dispatch experts style
e1498696
Docs: shorter token dispatch overview
e139e8a5
Replicate the rest of the plan under expert parallel token dispatch
cf74929d
Merge remote-tracking branch 'upstream/main' into fsdp2-ep-2d-mesh
0e21b598
Merge branch 'fsdp2-ep-2d-mesh' into ep-token-dispatch
18144abb
Update docs/source/en/expert_parallelism.md
c9a42167
Update docs/source/en/expert_parallelism.md
8f2f9cb1
Update docs/source/en/expert_parallelism.md
3c8f600e
Update docs/source/en/expert_parallelism.md
e7be30a9
Update docs/source/en/expert_parallelism.md
8dbccad4
Merge branch 'main' into fsdp2-ep-2d-mesh
35ed430b
Merge branch 'fsdp2-ep-2d-mesh' into ep-token-dispatch
79fe11d4
Keep replicated_with_grad_allreduce under expert-parallel dispatch
0817429f
Compare gradients, not only the loss, in the EP backward test
34e63c9d
Update docs/source/en/expert_parallelism.md
75fff716
Update docs/source/en/expert_parallelism.md
acba6bc9
Update docs/source/en/expert_parallelism.md
69668c3c
Update docs/source/en/expert_parallelism.md
9cfb2d3b
Merge branch 'main' into fsdp2-ep-2d-mesh
2884e2f1
Merge branch 'fsdp2-ep-2d-mesh' into ep-token-dispatch
4ce47004
Declare dispatch_experts_forward next to its only caller
28cb4827
Name the dispatch strategy instead of switching it with a bool
a6e36c09
Merge branch 'main' into fsdp2-ep-2d-mesh
1be3a7f5
Merge remote-tracking branch 'upstream/fsdp2-ep-2d-mesh' into ep-toke…
db788602
Merge branch 'main' into fsdp2-ep-2d-mesh
a6f5824a
Merge branch 'fsdp2-ep-2d-mesh' into ep-token-dispatch
b56e1d68
Merge branch 'main' into fsdp2-ep-2d-mesh
cb02b27f
Pass the ignored params through the policy helper
6a86e181
Merge remote-tracking branch 'upstream/main' into fsdp2-ep-2d-mesh
79988c35
Drop the comment on the parallel sizes
51207488
Trim the parallelism config comment
0147840f
Merge remote-tracking branch 'upstream/fsdp2-ep-2d-mesh' into ep-toke…
d604a2a2
Merge remote-tracking branch 'upstream/main' into ep-token-dispatch
b169a75d
Address review: guard dispatch + PP, derive the mesh name, move the g…
3439765f
Keep reading the tensor parallel size from accelerate in `get_tp_size`
4f2a6b91
Give the token dispatch weight comparison its own tolerance
86561b69
Separate expert dispatch mesh from tensor parallel size
df8fa771
Support trunk tensor parallelism with folded expert dispatch
60655f89
rm config test
d7753400
Simplify expert dispatch and rename edp to efsdp
a49aa6d1
Merge branch 'ep-token-dispatch' into ep-mesh
b1fdbdf3
Merge origin/ep-mesh and restore expert-parallel mesh support
6d8f54de
Unify dense and expert mesh construction
ea65cfb3
remove
b33e45e0
Encapsulate distributed mesh views behind named accessors
0eb9ffa8
Simplify MeshManager access and keep mesh views private
4d63a32c
Simplify expert parallel configuration and explicit plan selection
d4da6e0b
Resolve TP and EP plans before sharding and derive expert dispatch
6a1059ee
Merge branch 'main' into ep-token-dispatch
56047f4c
Merge branch 'ep-token-dispatch' into ep-mesh
00717373
Inline legacy expert parallel compatibility handling
c8a05bc6
Simplify parallel plan resolution and clarify TP and EP meshes
cc2237c5
remove
fc37645e
Remove auto EP plans and clarify parallel plan tests
251b2b94
Clarify expert parallel docs and explicit sharding mesh selection
8c2588da
Always wrap dispatched experts on the expert FSDP mesh
19118585
Clarify TP and EP layouts in FSDP comparison test
49f432cd
Clarify all-reduce and dispatch layouts in EP tests
6d231256
Take TP and EP meshes in apply_tensor_parallelism
e1d33d25
3outeille
commented
on 2026-09-15
3outeille
commented
on 2026-09-15
add default all-to-all plan for qwen3_moe only + derive rule from pl…
79fe4a43
raise in `resolve_parallel_plans` when an override key matches no mod…
2dc99524
better distinguish masked/all-to-all path for nicer API
b2b0d324
Login to write a write a comment.
Login via GitHub