transformers
[`distributed`]: adapt Trainer to work Expert Parallel
#48873
Merged

[`distributed`]: adapt Trainer to work Expert Parallel #48873

3outeille merged 113 commits into main from ep-trainer
3outeille
3outeille 3outeille changed the title Train models sharded at load time with the Trainer, token dispatch included (EP part 4) [`distributed`]: adapt Trainer to work Expert Parallel 66 days ago
HuggingFaceDocBuilderDev
qgallouedec
3outeille 3outeille force pushed from 8653dc4e to 3d529d5e 57 days ago
3outeille 3outeille force pushed from 3d529d5e to 580cb9f8 57 days ago
3outeille 3outeille marked this pull request as ready for review 57 days ago
github-actions github-actions requested a review from SunMarc SunMarc 57 days ago
github-actions github-actions requested a review from Rocketknight1 Rocketknight1 57 days ago
SunMarc
SunMarc commented on 2026-09-25
3outeille 3outeille force pushed from d32f7ef5 to 24be3cd7 54 days ago
3outeille
3outeille 3outeille force pushed from 026eb49b to 51b74d50 54 days ago
3outeille 3outeille removed review request from Rocketknight1 Rocketknight1 54 days ago
3outeille 3outeille force pushed from 51b74d50 to 040541a2 54 days ago
3outeille 3outeille force pushed from 6e9b1a41 to eacebd89 54 days ago
3outeille 3outeille force pushed from eacebd89 to 8a8b28b2 53 days ago
ArthurZucker
ArthurZucker approved these changes on 2026-09-30
SunMarc
SunMarc approved these changes on 2026-09-30
3outeille 3outeille force pushed from 8a8b28b2 to f7a128c8 52 days ago
3outeille 3outeille force pushed from f7a128c8 to 5c975a6e 52 days ago
3outeille 3outeille force pushed from 5c975a6e to 23905e8e 52 days ago
3outeille 3outeille force pushed from 23905e8e to c935d55e 52 days ago
3outeille Add dense and expert device mesh views with a MeshManager
eecb6a8a
3outeille cleaning
29f51267
3outeille clean
40745266
3outeille update
578af0e7
3outeille clean doc
083e2532
3outeille Use ep_size in the Kimi K2.5 expert parallel example
2e682969
3outeille Enable expert parallelism in the Mega MoE example
29eb5a36
3outeille warn once
d6090dfd
3outeille Add dense and expert device mesh views with a MeshManager
76ff6f0e
3outeille cleaning
9a1b525f
3outeille update
ce8657f9
3outeille clean doc
ea3aacba
3outeille Update src/transformers/distributed/mixin.py
d498e802
3outeille style: ruff import block formatting in distributed config
ce3f5760
3outeille add more comments
f8415e73
3outeille Decouple tp_plan and ep_plan for expert parallelism.
a115da8e
3outeille Match expert paths with a regex and keep only expert rules under toke…
271de167
3outeille revert change on warning
cc5200f9
3outeille Apply suggestion from @ArthurZucker
17f7b659
3outeille cleaning
ef837266
3outeille Fix tied embeddings for models with an EP-only base plan
e7bfac9b
3outeille [distributed] Add expert-parallel token dispatch, default for Qwen3 MoE
4cc84e42
3outeille clean
b756980d
3outeille fix
4d5ef4cf
3outeille make check repo
556f4b9e
3outeille Drop the load-time ep_size == tp_size guard that rejects token dispatch
59578e01
3outeille remove all to all warning by using functional version
16fbcca9
3outeille use ep plan instead of group_gemm is_expert
60fd0822
3outeille use dtensor instead of manuall all_reduce
fc42fb9c
3outeille better comment
1478d288
3outeille [`distributed`]: Fix DeepSeek-V4 tied embeddings with combined TP and…
51472f93
3outeille 3outeille force pushed from c935d55e to a2671dcc 51 days ago
3outeille linting
ea814ebb
3outeille Add dense and expert device mesh views with a MeshManager
925415d4
3outeille cleaning
0c227f1d
3outeille clean
a5958d4e
3outeille update
65719164
3outeille clean doc
90fbd095
3outeille Use ep_size in the Kimi K2.5 expert parallel example
e4561f09
3outeille Enable expert parallelism in the Mega MoE example
7f4951f9
3outeille warn once
e65e3eaf
3outeille Add dense and expert device mesh views with a MeshManager
03af9e8b
3outeille cleaning
69a145b2
3outeille update
caa76409
3outeille clean doc
a3de4dc2
3outeille Update src/transformers/distributed/mixin.py
84cb4bee
3outeille style: ruff import block formatting in distributed config
b5521952
3outeille add more comments
d1cf2079
3outeille Decouple tp_plan and ep_plan for expert parallelism.
4b655fbf
3outeille Match expert paths with a regex and keep only expert rules under toke…
a4f11e3a
3outeille revert change on warning
a5cd182b
3outeille Apply suggestion from @ArthurZucker
3d45692b
3outeille cleaning
04752eb4
3outeille Fix tied embeddings for models with an EP-only base plan
c41435b0
3outeille [distributed] Add expert-parallel token dispatch, default for Qwen3 MoE
8dc8333b
3outeille clean
865d5d96
3outeille fix
cac1b2b0
3outeille make check repo
c59446d9
3outeille Drop the load-time ep_size == tp_size guard that rejects token dispatch
abf336cc
3outeille remove all to all warning by using functional version
6244a1c3
3outeille use ep plan instead of group_gemm is_expert
c72bbf72
3outeille use dtensor instead of manuall all_reduce
c31a1b7a
3outeille better comment
00ee316a
3outeille linting
ec8278e1
3outeille Train models sharded at load time with the Trainer, token dispatch in…
39c7e413
3outeille Use ep-dispatch version of expert_parallelism.md
dfec8d1e
3outeille Sync replicated trainable parameters when training a model sharded at…
628da80a
3outeille Use ep-dispatch version of expert_parallelism.md
dfec8d1e
3outeille Fix utf-8 encoding in expert parallel trainer tests
e72d719b
3outeille use attribute instead
69ecbf13
3outeille 3outeille force pushed from a2671dcc to 69ecbf13 47 days ago
3outeille fix
bcbb851d
3outeille Merge branch 'ep-mesh-infrastructure' of github.com:huggingface/trans…
d51aca88
3outeille Merge branch 'ep-plans' of github.com:huggingface/transformers into e…
448315ed
3outeille revert
877afeea
3outeille Merge branch 'ep-dispatch' of github.com:huggingface/transformers int…
bec5a12b
3outeille Merge branch 'main' into ep-mesh-infrastructure
0ec824f1
3outeille Merge branch 'ep-plans' into ep-dispatch
b20486ce
3outeille Merge branch 'ep-mesh-infrastructure' into ep-plans
c85c8bca
3outeille revert
0d8d4f55
3outeille Merge branch 'ep-dispatch' into ep-trainer
a7a37bed
3outeille Merge branch 'ep-plans' into ep-dispatch
c9d64290
3outeille typo
cedde47c
3outeille Merge branch 'main' into ep-mesh-infrastructure
b62bbbcd
3outeille clearer
6a23a306
3outeille Merge branch 'ep-mesh-infrastructure' of github.com:huggingface/trans…
daac87a5
3outeille Merge branch 'main' into ep-mesh-infrastructure
6c73628e
3outeille clean
b4d81b5b
3outeille Merge branch 'ep-mesh-infrastructure' of github.com:huggingface/trans…
e3cc578e
3outeille Merge branch 'ep-mesh-infrastructure' into ep-plans
9bb2518a
3outeille Merge branch 'ep-plans' into ep-dispatch
38704ae8
3outeille cleaning
7b43249c
3outeille cleaning
e0f70125
3outeille Merge branch 'ep-plans' of github.com:huggingface/transformers into e…
926ee709
3outeille Merge branch 'main' into ep-mesh-infrastructure
746b2c53
3outeille Merge branch 'ep-mesh-infrastructure' into ep-plans
0eac0f5c
3outeille Merge branch 'ep-plans' into ep-dispatch
030900b0
3outeille dont torch cat for empty tokens
af793976
3outeille Merge branch 'ep-dispatch' of github.com:huggingface/transformers int…
43a8c2d6
3outeille Merge branch 'ep-dispatch' into ep-trainer
37089c18
3outeille check code quality
865aa963
3outeille Update src/transformers/distributed/utils.py
7b57dd23
3outeille Merge branch 'main' into ep-mesh-infrastructure
ec77a598
3outeille Update src/transformers/distributed/configuration_utils.py
7af3cbc9
3outeille linting
f37bc0f4
3outeille Merge branch 'ep-mesh-infrastructure' of github.com:huggingface/trans…
11f0a3d5
3outeille fix comment
a4d06939
3outeille Merge branch 'ep-mesh-infrastructure' into ep-plans
6a504c18
3outeille Merge branch 'main' into ep-mesh-infrastructure
812c060c
3outeille Merge branch 'ep-plans' into ep-dispatch
13d2fc74
3outeille Merge branch 'ep-mesh-infrastructure' into ep-plans
430bb2e6
3outeille Merge branch 'ep-dispatch' into ep-trainer
e431d5cb
3outeille Merge branch 'ep-plans' into ep-dispatch
42d06077
3outeille Merge branch 'ep-dispatch' into ep-trainer
b0fd271d
Base automatically changed from ep-dispatch to main 46 days ago
3outeille Merge branch 'main' into ep-trainer
007a95fe
3outeille 3outeille enabled auto-merge 46 days ago
github-actions
3outeille 3outeille merged 8625785e into main 46 days ago
3outeille 3outeille deleted the ep-trainer branch 46 days ago
mgzbpfrnkr-svg
mgzbpfrnkr-svg commented on 2026-10-08
mgzbpfrnkr-svg

Login to write a write a comment.

Login via GitHub

Assignees
No one assigned
Labels
Milestone