transformers
Shard the parameters kept in float32 in their own FSDP2 group
#49501
Merged
Merged
Shard the parameters kept in float32 in their own FSDP2 group #49501
3outeille
merged
41 commits into
main
from
fsdp-fp32-param-groups
FSDP2: shard the parameters kept in float32 in their own group
8c1d1156
Test a bf16 training step under FSDP2, and run the FSDP tests on ERNI…
3623fdfd
ArthurZucker
commented
on 2026-10-12
build and expand once the tp/ep_plan
6440b726
unify
79b5fdb0
clean fsdp
bb63397e
reorganize to avoid breaking changes
f9158f8d
better distinction
64eb317a
Merge branch 'main' into unify-resolve-plan
61a4954d
revert
5fbf21c9
Merge branch 'unify-resolve-plan' of github.com:huggingface/transform…
ff70527c
add tp plan to deepseek v4 (but attention stays replicated)
3f1e4ab9
Revert "add tp plan to deepseek v4 (but attention stays replicated)"
ff3629c6
Merge branch 'main' into unify-resolve-plan
b5da03f9
Fix the merge of main: restore moved imports and the empty tp_plan ov…
d527c29f
Merge remote-tracking branch 'origin/main' into unify-resolve-plan
523101d8
cleaner mixin
96332a1e
cleaner
615fe835
deprecation cycle
868647f2
cleaner
a121baea
revert tp plan
e1d83ef5
Merge remote-tracking branch 'origin/main' into unify-resolve-plan
2dbcb42c
Pass the expanded plan in the bare-module colwise_gather_output test
9ef53bbc
retrigger to verify fetcher
a1a055ba
the branch is now pointing to main and not to unify-resolve-plan
b5eb8827
Refactor expert parallelism registration and improve documentation
0de36597
comment
bc4b6e0a
some models missed the tests EP mixin
c47e5dce
fix copyright
f88b6167
Merge branch 'cleanup-ep' into unify-resolve-plan
7d3345d8
move replace_layer_number_by_wildcard and _get_parameter_plan to para…
da4eed4d
Merge branch 'main' into cleanup-ep
d7591e12
Merge branch 'cleanup-ep' into unify-resolve-plan
a75a5cb2
Merge branch 'main' into unify-resolve-plan
6247bd14
Merge remote-tracking branch 'origin/unify-resolve-plan' into unify-r…
f5824079
Merge remote-tracking branch 'origin/unify-resolve-plan' into fsdp-fp…
1e3bd8e5
Group the float32-kept parameters while resolving the FSDP plan
d35cda87
Find the float32 modules from the names `_resolve_parallel_plans` alr…
27c38f45
Initialize the ERNIE-4.5-MoE router, and skip parameters without a gr…
639ac786
Base automatically changed from
unify-resolve-plan
to
main
3 days ago
Merge remote-tracking branch 'origin/main' into fsdp-fp32-param-groups
d0bb814e
Reword the comment on the float32 FSDP2 groups
aa09bc5d
Swap the words in the float32 FSDP2 group comment
66342ec0
Login to write a write a comment.
Login via GitHub