DeepSpeed
[AutoSP] (Sequence Parallelism) support for Multimodal Models (ViT + LLM)
#7984
Merged
Merged
[AutoSP] (Sequence Parallelism) support for Multimodal Models (ViT + LLM) #7984
delock
merged
35 commits into
deepspeedai:master
from
nathon-lee:multimodal-seq-parallel
Initial plan
001f77c3
Revert "fix: update 1 file reformatted."
b90aee5a
Merge pull request #5 from nathon-lee/copilot/git-revert-ff886701
b6da9afd
Merge branch 'deepspeedai:master' into master
bb7f64fd
Initial plan
cbc816c9
Reapply "fix: update 1 file reformatted."
5fcc9a7e
Merge pull request #6 from nathon-lee/copilot/remove-commits-from-master
f7c5d75d
Merge branch 'deepspeedai:master' into master
18efbcc3
Merge branch 'deepspeedai:master' into master
e2ac74d2
Merge branch 'deepspeedai:master' into master
da07382d
Merge branch 'deepspeedai:master' into master
5d8875cc
Merge branch 'deepspeedai:master' into master
316b6dda
Merge branch 'deepspeedai:master' into master
2020543f
Merge branch 'deepspeedai:master' into master
1a8694c6
Merge branch 'deepspeedai:master' into master
d6725be0
feat: Add AutoSP scaffolding for multimodal sequence parallelism
a0788ea1
fix: fix some format issue by pre-commit
0c6c14c9
tests: Add tests for ModalityFusionSPAdapter and LLM wrap path
ac71ca96
fix: fix some format issue by pre-commit
bb027587
[Sequence Parallelism] Implement LlavaFusionAdapter for visual token …
efc95604
feat: Add numerical equivalence tests for AutoSP multimodal SP
5aa270b1
test: add numerical equivalence tests for AutoSP multimodal sequence …
d7de96fc
test: fix check-torchcuda CI failure in autosp tests
bc01ed75
feat: Added two new adapter classes after LlavaFusionAdapter
34ca8178
[AutoSP] Add InternVLFusionAdapter and Qwen2VLFusionAdapter
2fd81bcd
[AutoSP] Add integration tests and multimodal SP benchmark
6bf1ec31
fix: fix some warn errs .
4e7726e2
fix: benchmarks/autosp: replace torch.cuda with get_accelerator()
c21fe99b
delock
commented
on 2026-04-28
delock
commented
on 2026-04-28
[AutoSP] Fix ViT CLS handling and skip incompatible HF LLM wrapping
baa8e510
fix: Fix non-divisible patch padding in UlyssesSPViTAttention; add do…
b46e5888
fix: fix some format errs by tool
9f0c277b
Merge pull request #15 from nathon-lee/multimodal-seq-parallel_bck
8bfc1ed9
delock
commented
on 2026-04-29
[AutoSP] Fix padding-before-attention bug in UlyssesSPViTAttention
b6bb8b3c
Merge pull request #16 from nathon-lee/multimodal-seq-parallel_bck
462f1697
delock
approved these changes
on 2026-05-01
Merge branch 'master' into multimodal-seq-parallel
8055a03c
Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub