[Trainer] accelerate contextparallel support in trainer #40205
SunMarc
merged
33 commits into
main
from
trainer-cp
initial context_parallel_size support in trainer
ca8e3663
Merge branch 'main' into trainer-cp
29b22cf7
SalmanMohammadi
commented
on 2025-08-15
SalmanMohammadi
commented
on 2025-08-15
For context parallelism, use AVG instead of SUM to avoid over-account…
d70fef40
git pushMerge branch 'trainer-cp' of https://github.com/huggingface/t…
dd58dd01
use parallelism_config.cp_enabled
ecc23661
add parallelism_config to trainer state
a629ff0a
winglian
commented
on 2025-08-15
SalmanMohammadi
commented
on 2025-08-15
warn when auto-enabling FSDP
66d4273e
S1ro1
commented
on 2025-08-15
S1ro1
commented
on 2025-08-15
S1ro1
commented
on 2025-08-15
S1ro1
commented
on 2025-08-15
S1ro1
commented
on 2025-08-15
fix some reviews
ffa4699c
WIP: somewhat matching loss
361f122f
S1ro1
commented
on 2025-08-18
S1ro1
commented
on 2025-08-18
S1ro1
commented
on 2025-08-18
S1ro1
commented
on 2025-08-18
Merge branch 'main' into trainer-cp
3d426c16
Merge branch 'main' into trainer-cp
3efe69bf
Feat: add back nested_gather
eca52ac0
Feat: cleanup
951527b1
Fix: raise on non-sdpa attn
412c15e2
Merge branch 'main' into trainer-cp
be60c40f
remove context_parallel_size from TrainingArguments
2c357aab
S1ro1
commented
on 2025-08-19
if we have parallelism_config, we defer to get_state_dict from accele…
71e082f8
Merge branch 'main' into trainer-cp
37e6fdfe
Merge branch 'main' into trainer-cp
4f6fe153
fix form review
485d7fab
S1ro1
commented
on 2025-08-22
Feat: add parallelism config support
3d16def7
Chore: revert some unwanted formatting changes
25a308e8
Fix: check None
6d413651
Check none 2
d82022c9
Fix: remove duplicate import
ae9f878b
Merge branch 'main' into trainer-cp
531924e2
Merge branch 'main' into trainer-cp
64d73362
Update src/transformers/trainer.py
52cb3bc9
Update src/transformers/training_args.py
6e9fb308
Merge branch 'main' into trainer-cp
bf187b22
Fin
33817f37
SunMarc
approved these changes
on 2025-08-25
SunMarc
commented
on 2025-08-25
require accerelate 1.10.1 and higer
22945068
SunMarc
approved these changes
on 2025-08-26
Merge branch 'main' into trainer-cp
f956348d
Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub