Deepspeed Ulysses/ALST integration (#3817)
* Feat: initial impl
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* improve
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* s/flavour/backend/
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* style + ver
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* better check
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* check
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* docs + example
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* add tests
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* add tests
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* cleanup
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* cleanup
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* Apply suggestions from code review
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* add experimental notice
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* style
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* new deepspeed version
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* additional checks + tests
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* more docs
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* more docs
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* working now
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* style
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* update docs
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* more robust config parsing
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* fix
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* Apply suggestions from code review
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* check backend, integrate ulysses API improvement
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* style
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* fix default to match the doc
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* Apply suggestions from code review
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* fix
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* deepspeed=0.18.2 is out
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* Apply suggestions from code review
Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>
* s/cp/sp
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* fixes
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* Apply suggestions from code review
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* Update src/accelerate/parallelism_config.py
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* suggestion
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* Update docs/source/concept_guides/sequence_parallelism.md
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* Update sequence_parallelism.md
* fix
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* fix
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* fix
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
* Apply suggestion from @kashif
* Apply suggestion from @kashif
* Apply suggestion from @kashif
* Apply suggestion from @kashif
* Apply suggestion from @kashif
* Apply suggestion from @kashif
* Apply suggestion from @kashif
* Apply suggestion from @kashif
* Apply suggestion from @kashif
* Apply suggestion from @kashif
* Apply suggestion from @kashif
---------
Signed-off-by: Stas Bekman <stas.bekman@snowflake.com>
Co-authored-by: S1ro1 <matej.sirovatka@gmail.com>
Co-authored-by: Stas Bekman <stas.bekman@snowflake.com>
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>
