Add FSDP plans to all models #47165
3outeille
merged
194 commits into
main
from
split/a-pr-5-fsdp-plans
add distributed config
799ac94d
Add native FSDP2 module and migrate FSDP imports (Phase A PR-2).
22d4b52b
linting
4bfd1a64
unecessary
9487bdda
copyright edit
588884e6
revert
8cc48a06
add shard on read
672bbbe0
Merge branch 'main' into split/a-pr-1-distributed-config
5bbc7960
jsut shard on read
d0cc6a7d
Merge branch 'split/a-pr-1-distributed-config' into split/a-pr-2-fsdp…
6fd78131
Merge branch 'main' into split/a-pr-3-dual-path-loading
47519d7b
cleaning
063e57b0
Merge branch 'split/a-pr-3-dual-path-loading' of https://github.com/h…
c06dc35f
linting
9d2e86dd
fix
79457b3f
fix
f3e80210
Merge branch 'split/a-pr-1-distributed-config' into split/a-pr-2-fsdp…
acacae84
Merge branch 'main' into split/a-pr-1-distributed-config
54c1f4e5
Merge branch 'main' into split/a-pr-3-dual-path-loading
d23986ac
Merge branch 'split/a-pr-1-distributed-config' into split/a-pr-2-fsdp…
f219c74c
remove redundant test file
ea8243f8
Merge branch 'main' into split/a-pr-1-distributed-config
db31b043
Merge branch 'split/a-pr-1-distributed-config' into split/a-pr-2-fsdp…
4d840dcc
Update src/transformers/distributed/fsdp.py
c384fcdc
avoid looping, just look at dict
96258163
expand_fsdp returns reshard_targets, no_reshard_targets right away
59bcec56
better _resolve_tied_embed_lm_head_plan
ebf3585c
cleaning
e9693251
ruff
2376965e
Merge branch 'main' into split/a-pr-1-distributed-config
d8301145
Merge branch 'split/a-pr-1-distributed-config' into split/a-pr-2-fsdp…
a44f81f2
more robust detection of embed and lm_head
0f62c45e
Merge branch 'split/a-pr-2-fsdp-module' of https://github.com/hugging…
020f7d3f
Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-l…
05d766f5
cleaning
da302ade
ruff
dfc665cd
typo
446fd6e9
cleaner
5aeaff74
cleaner
819ff149
Merge branch 'main' into split/a-pr-1-distributed-config
413d775c
Merge branch 'split/a-pr-1-distributed-config' into split/a-pr-2-fsdp…
7bc37223
Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-l…
d6d8b51b
Merge branch 'main' into split/a-pr-2-fsdp-module
f8f27ffd
Merge branch 'main' into split/a-pr-2-fsdp-module
ce2f0019
Merge branch 'main' into split/a-pr-2-fsdp-module
ec87fff3
Merge branch 'main' into split/a-pr-2-fsdp-module
6e156fd4
Merge branch 'main' into split/a-pr-2-fsdp-module
606df0ac
typo
3ee8a928
Merge branch 'split/a-pr-3-dual-path-loading' of https://github.com/h…
6bc43aed
Merge branch 'main' into split/a-pr-2-fsdp-module
c4aa4b70
Merge branch 'main' into split/a-pr-2-fsdp-module
c5ad67b5
refactor dense path + apply_contiguous_shard
3c6c5f23
linting
74490d66
cleaning
db6c99e8
Merge branch 'main' into split/a-pr-2-fsdp-module
04c124ed
refactor _apply_strided_shard
8f214ef6
better
ab34afb2
refactor _slice_and_cat
c4bf166f
better comment
be61c49a
refactor moe dtensor shard ops
75ad5b9d
better comment
91cd7977
comment
68069dec
cleaning
83029ed7
linting
1f857df7
Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-l…
aa140ed7
Merge branch 'main' into split/a-pr-2-fsdp-module
8b57aa41
Add FSDP orchestration: mesh init, distribute-before-load, and DCP save.
7c113940
Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-l…
70544cd9
Merge branch 'split/a-pr-3-dual-path-loading' into split/a-pr-4-fsdp-…
17c6d402
add fsdp plan to 2 models for now
00eb1166
add tests fsdp mixin
be296ddf
linting
05900d60
refactor test fsdp mixin
fc2423bd
test fsdp mixin cleaning
5bbd8201
remove fsdp policy in tests + trim down further
b6d0b67a
test fsdp clean
ea361231
restore test_modeling_utils
bec4d23a
linting
8d3d3298
start trim down stuff
6316ee1e
fix
6e9004ec
breaking: cleaning modeling_utils.py
68df491a
load path with fsdp (dtensor) and tp (old tp) is linked
16b0b291
linting
e976a44c
add saving
a2fb155f
styling
5f52f196
fix tp ci
7f543017
add fsdp to ci
99f79acc
linting
b4514906
pick one model only for this PR
54ff4d1d
restore
33995399
trigger fsdp ci
11cf79c2
doc cleaning + tp_size remove
5b7ac3e5
fix tp ci for ep
06b0c394
edit doc
7a94d77a
move distributed function to utils + guarding
f3c742b7
linting
37df13ee
Merge branch 'main' into split/a-pr-2-fsdp-module
dd1000bb
expand_fsdp_plan iterate over modules
5c95559e
Merge branch 'split/a-pr-2-fsdp-module' of https://github.com/hugging…
00a11b6a
Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-l…
fd3d4b08
Merge branch 'split/a-pr-3-dual-path-loading' into split/a-pr-4-fsdp-…
86875d22
comment about tie embedding
e4613e67
Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-l…
c2b8451a
add comment tied embedding
d558f999
Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-l…
aed1818b
Merge branch 'split/a-pr-3-dual-path-loading' into split/a-pr-4-fsdp-…
450579ba
add DistributedMixin
91260ba4
Merge branch 'main' into split/a-pr-2-fsdp-module
1541674d
Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-l…
9fd759d8
solve merge conflict
84dd1078
Merge branch 'main' into split/a-pr-2-fsdp-module
9fae1b40
some cleaning
eb653bda
Merge branch 'split/a-pr-3-dual-path-loading' of https://github.com/h…
dcaa20ae
cleaning + comment
d03499fa
rename function for clarity
fb5533e1
Apply suggestion from @ArthurZucker
7838dc6c
doc
877052b5
Merge branch 'split/a-pr-3-dual-path-loading' of https://github.com/h…
59f666fa
comment
17643a7b
Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-l…
7bb152cd
Merge branch 'main' into split/a-pr-2-fsdp-module
0b4d1e06
Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-l…
101562d0
Merge branch 'split/a-pr-3-dual-path-loading' into split/a-pr-4-fsdp-…
99576ee8
linting
22215b7c
Merge branch 'split/a-pr-4-fsdp-orchestration' of https://github.com/…
f6ed4657
refactor
18d709c6
abstract to mixin
566ef6e4
typo
b632db38
Add FSDP plans to all models from distributed branch.
0eec8c78
fsdp plans
a1f4b71e
add fsdp plans
a07e5bf8
linting
0dfe224d
linting
e5a65357
Add distributed runtime utils and DistributedMixin (FSDP orchestratio…
e764cebd
Wire DistributedConfig through from_pretrained and save_pretrained (F…
61362b74
Add FSDP CI and end-to-end FSDP tests (FSDP orchestration 3/3).
eb34d37f
addd ep_plan
b866f008
Merge branch 'split/a-pr-4-1-distributed-runtime' into split/a-pr-4-2…
3da6fb9a
Merge branch 'split/a-pr-4-2-orchestration' into split/a-pr-4-3-fsdp-…
ca8d0370
restore validate module
b3f45385
Wire DistributedConfig through from_pretrained and save_pretrained.
b1fb1b93
revert
1bf32cc4
Merge branch 'split/a-pr-4-1-distributed-runtime' into split/a-pr-4-2…
1620c0f2
Merge branch 'split/a-pr-4-2-orchestration' into split/a-pr-4-3-fsdp-…
fe0dcf7d
inline distribute_model
e52fda3f
revert
71dde8aa
remove saving/loading
4e23dbf1
leaner mixin
5742a2ef
downgrade torch version guarding
5c81035d
remove
004354f8
linting
6d03414a
Merge branch 'split/a-pr-4-1-distributed-runtime' into split/a-pr-4-3…
450987f3
revert
47d85adb
Merge branch 'split/a-pr-4-2-fsdp-tests-ci' of https://github.com/hug…
c577f337
revert
b319d6ac
post_init() parallel plan move to mixin
eb9a081e
Merge branch 'split/a-pr-4-1-distributed-runtime' into split/a-pr-4-2…
d5358f3e
revert tp mixin
9a783c8c
Merge branch 'split/a-pr-4-1-distributed-runtime' into split/a-pr-4-2…
0448e72f
add save/load
259bff43
Merge branch 'main' into split/a-pr-4-1-distributed-runtime
b15b63e1
Merge branch 'main' into split/a-pr-4-1-distributed-runtime
17d611b4
Merge branch 'main' into split/a-pr-4-1-distributed-runtime
7fe841b7
Merge branch 'split/a-pr-4-1-distributed-runtime' into split/a-pr-4-2…
5f9b5ab2
Merge branch 'main' into split/a-pr-4-2-fsdp-tests-ci
b95be32b
3outeille
changed the base branch from
split/a-pr-4-fsdp-orchestration
to
split/a-pr-4-2-fsdp-tests-ci
35 days ago
only FSDP save/load for now
d6257574
Merge branch 'split/a-pr-4-2-fsdp-tests-ci' of https://github.com/hug…
9e810e40
revert
049b89c0
Merge remote-tracking branch 'origin/split/a-pr-4-2-fsdp-tests-ci' in…
a03f35a0
refactor
6b35d0e3
Merge branch 'split/a-pr-4-2-fsdp-tests-ci' into split/a-pr-5-fsdp-plans
862b428b
modular
e5032053
Merge branch 'split/a-pr-5-fsdp-plans' of https://github.com/huggingf…
c7fa8437
Merge branch 'main' into split/a-pr-4-2-fsdp-tests-ci
047bd41d
Merge branch 'main' into split/a-pr-4-2-fsdp-tests-ci
8c96054e
Merge branch 'split/a-pr-4-2-fsdp-tests-ci' into split/a-pr-5-fsdp-plans
a1b0d779
Base automatically changed from
split/a-pr-4-2-fsdp-tests-ci
to
main
33 days ago
Merge branch 'main' into split/a-pr-5-fsdp-plans
c364a541
ea
36ca137e
Merge branch 'main' into split/a-pr-5-fsdp-plans
2f87b7db
Merge branch 'main' into split/a-pr-5-fsdp-plans
d44d5db9
add it to pretrained model
507a3c7d
linting
258ab19f
fix the test by moving in init class the fsdp plan instead of post init
70de2959
edit
830f0e40
Merge branch 'main' into split/a-pr-5-fsdp-plans
e1880b19
ArthurZucker
approved these changes
on 2026-07-27
Merge branch 'main' into split/a-pr-5-fsdp-plans
7a4f465a
revert unwanted changes
3d7d2f2d
pull
6232a5e8
Merge branch 'main' into split/a-pr-5-fsdp-plans
c277d336
Merge branch 'main' into split/a-pr-5-fsdp-plans
a23b171d
Merge branch 'main' into split/a-pr-5-fsdp-plans
f9273afb
Merge branch 'main' into split/a-pr-5-fsdp-plans
e63e45c3
fix ci
47e72898
Merge branch 'split/a-pr-5-fsdp-plans' of https://github.com/huggingf…
6c5f6c02
Login to write a write a comment.
Login via GitHub