accelerate
Parallelism config + TP + HSDP + BYODM (Bring Your Own Device Mesh)
#3682
Merged
Merged
Parallelism config + TP + HSDP + BYODM (Bring Your Own Device Mesh) #3682
S1ro1
merged
76 commits into
huggingface:main
from
salmanmohammadi:device_mesh_parallelism_config
Feat: init
2f471e3a
Feat: add validation + init from kwargs
43b1ca7a
Fix: minor fixes
79faa138
Feat: more cleanup
16f348bb
Minor refactor
53ef5247
remove import
cd31b02b
adding support for pre-configured device mesh
2d892105
salmanmohammadi
commented
on 2025-07-16
salmanmohammadi
commented
on 2025-07-16
adding device mesh to fsdp2
afaafef6
salmanmohammadi
commented
on 2025-07-16
moving mesh dim defn to parralismconfig
2d952cba
tests
91ca626f
WIP device mesh/accelerator validation
910368b8
WIP more tests
b7d154ec
Test Driven Development (TDD)
8a0de72b
fixing build_device_mesh
1c68efb4
FSDP dim names
e01abf13
adding example
69b523c6
WIP
c765a447
salmanmohammadi
changed the title
fixing HSDP
8d97930c
Feat: add back old options
57c0d9e0
working example
c93285a4
winglian
commented
on 2025-07-21
winglian
commented
on 2025-07-21
debugging
cb40d36d
adding parallelism config to partialstate
b76ee67a
Feat: revert ddp changes
9aa26123
Revert DDP
de96e74d
Feat: (untested) update mesh dims and some minor tweaks
fd05e3bc
adding dp_cp dims
efc903e0
updating comments
7c3d0e3c
winglian
commented
on 2025-07-22
S1ro1
commented
on 2025-07-22
WIP
3cfce252
wip 2
1bbdb751
reverting
aa749ad3
storing state in accelerator rather than acceleratorstate
aa745766
Fix: minor tweaks
4e99b9cd
wip example update
3d235cb4
merging
61868c29
Fixes for non-fsdp2 case
f96fea3c
Feat: ensure ddp/tp only works
dd894525
winglian
commented
on 2025-07-23
winglian
commented
on 2025-07-23
salmanmohammadi
commented
on 2025-07-23
updating example
7f243e09
updating example
4a2dd58f
updating examples, fixing state
dc145c2b
salmanmohammadi
commented
on 2025-07-23
fixed state
f21547f3
comments
1a49c164
fixing partial state check
07bf2b3b
linting
f274b354
salmanmohammadi
changed the title
comments
a6feca96
removing fn
80deb7ee
merging
52c178fe
winglian
commented
on 2025-07-23
winglian
commented
on 2025-07-23
WIP: fix tp
133ef5f7
S1ro1
commented
on 2025-07-23
winglian
commented
on 2025-07-24
comments
74009ea7
removing return
379daa0b
reverting upcast
168b5202
winglian
commented
on 2025-07-24
add guards
76a546fd
guards for empty self.parallelism_config
e8963dc1
salmanmohammadi
commented
on 2025-07-25
salmanmohammadi
commented
on 2025-07-25
salmanmohammadi
commented
on 2025-07-25
salmanmohammadi
commented
on 2025-07-25
salmanmohammadi
commented
on 2025-07-25
salmanmohammadi
commented
on 2025-07-25
use len on tuple to check if empty
a402faff
Feat: cleanup example
235d29ff
Feat: some cleanup of example
1017752a
Merge branch 'main' into device_mesh_parallelism_config
36a1234c
Feat: add trackio
7ddb3abb
Fix: improve trackio
9fdc320d
Feat: TP works
00dd4af6
Feat: some fsdp2 improv
d21ff9f2
Feat: working examples
d2608422
S1ro1
commented
on 2025-07-28
handle clipping for tensor parallel
8b89d278
Implicit replicate
4709fc88
SunMarc
commented
on 2025-07-29
Refactor: move to separate file + cleanup + basic comments
353b5593
Fix: add unadded files, fix circular import
7364440b
Feat: better readme
e90f832c
Feat: add blog + ultrascale links
044c7130
Tmp: should_save_model now returns only true
464a642d
Fix: remove implicit_replication and style
f85eadf1
Fix: remove optional
86771e25
add guard on parallelism_config.tp_enabled
c80aae08
fix import
c8a2ae56
fixing empty parallelism_config
ec59f84d
fix import path for test patch
0afb69f1
fixing patch
89aad7a3
merging
c570f7c3
Login to write a write a comment.
Login via GitHub