LoRA and Transformers TP #3079
BenjaminBossan
merged
36 commits into
huggingface:main
from
michaelbenayoun:lora_and_tp
wip(tp): add hooks to LoRA adapters for TP
e90907a3
feat: add hooks to LoRA adapters for TP plan
17b96ee1
wip: shard LoRA adapters for TP
8e8ea685
wip: add TP hooks to adapters
a4806952
feat: add hooks for TP in LoraModel
744a0a53
fix: add lora adapter weight broadcasting after initialization
d934e4c9
style: add comments and remove space
99260d18
fix: load and shard from checkpoints with TP
03c94fb6
test: add test suites for LoRA + Transformers TP
69e80296
refactor: rename test
67d1b392
refactor: rename test file
7000ae84
style: fix length
4cb247b4
test: add overfitting test
e9536e94
fix: remove comma and unnecessary if statement
1c6d4a06
style: torch.distributed to dist
25e7926d
Merge branch 'main' into lora_and_tp
4ce5ba38
fix: move adapters to device before broadcast
38865ff2
fix: check for correct transformers version
dbcbbddd
fix: lazy import to avoid failing with older transformers versions
fe773497
doc: mention TP support in LoRA docs
07daec84
style: extend comment on empty_param
786fceb7
refactor: remove duplicated code
4fa66822
refactor: remove duplicated code when adding the hooks
ed5e33ce
style: ruff format
89b3f35d
test: move tp tests to gpu
0d238ec4
test: remove test file
fb0e31a8
fix: typos and arguments
e254738e
fix: restore pyproject.toml
32eeb5a7
fix: tests on GPUs
040111e6
fix: use internal model
c3d15fb4
fix: restore pyproject.toml
75c2f3b7
fix: change repo name to internal
14a29722
test: launch tp training integration test
749ee5bb
refactor: add decorator to the test class
26fa93f5
Merge branch 'main' into lora_and_tp
7f14a565
BenjaminBossan
commented
on 2026-03-18
fix: add missing config file
ecb29ed4
BenjaminBossan
approved these changes
on 2026-03-18
Login to write a write a comment.
Login via GitHub