DeepSpeed
Long sequence parallelism (Ulysses) integration with HuggingFace
#5774
Merged
Merged
Long sequence parallelism (Ulysses) integration with HuggingFace #5774
loadams
merged
30 commits into
deepspeedai:master
from
samadejacobs:uly-hf
Use PyTorch mesh tensor to create grps
97e8b413
Add sequence parallel grp and test example scripts
e4a42c08
Remove debug statement
b2e09aa0
Set seq parallel group in DS engine
8b9a25fc
Pass mesh device to DS config
02b97db9
Add sp CE
c9f2505d
Remove debug statment
69f16c3f
Set flash attn as required and required torch version
d79107a9
Clean up debug statement, clarify log msg
dcbf8699
Remove deprecated test file
a4044529
Remove deprecated test bash script
a2110782
Merge remote-tracking branch 'origin/master' into uly-hf
453cbee7
Code clean up, fix where batch first in all2all
c5ba5261
BacharL
commented
on 2024-07-21
tohtana
commented
on 2024-07-22
tohtana
commented
on 2024-07-22
Delete tests/small_model_debugging/test_ulysses_hf.py
00b05f89
Fix sp in ds engine to align with new (latest) HF transformer flash_attn
2f4bd5a1
Warn is backend does not support mesh device, fail fast if seq parall…
fc119ca4
Repalce cuda with get_accelerator, set minimum torch version for devi…
d4b19d10
Merge branch 'master' into uly-hf
073c3817
Add unit tests
686a1840
Merge branch 'uly-hf' of https://github.com/samadejacobs/DeepSpeed in…
1a22cd16
Add support to (optionally) set device mesh config (dp_size, sp_size)…
fc9330d6
Merge branch 'master' into uly-hf
7a59d3c5
Merge branch 'master' into uly-hf
aaa79eb8
Add (now/newly required) batch_dim_idx to unit test
43806b48
costin-eseanu
approved these changes
on 2024-08-09
Fix formating
f4bd5f42
Merge branch 'master' into uly-hf
7a643362
disabled auto-merge 1 year ago
Manually disabled by user
Manually disabled by user
Merge branch 'master' into uly-hf
14f51ad9
Merge branch 'master' into uly-hf
76c67c04
Update test_ulysses.py
cb7c20e6
Merge branch 'master' into uly-hf
513d479e
Login to write a write a comment.
Login via GitHub