POC: multiple model/configuration DeepSpeed support (#3097)
* Bookmark
* Migratory
* Uncomment
* Rm name to model for now
* Rm container
* Left: test
* Allow only wrapping one model
* Add warning but only ref once
* Refine
* Update src/accelerate/accelerator.py
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
* Finish stas nits
* Clean
* Fixup test + test writing
* Fully working
* Fin
* Nit
* Quality
* Update src/accelerate/accelerator.py
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
* Actionable error
* Make note of when its enabled
* Apply suggestions from code review
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
* Merge tests
* Merge
* Add currently broken test script
* Push the working implementation
* Fin
* Add guards for user behavior
* Test nits
* TODO: finish knowledge distillation example
* Update tests/deepspeed/test_deepspeed_multiple_model.py
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
* Allow for dict-like interface
* Get rid of disable
* Uncomment
* Complete rewrite to force a dict to be used
* Working tests/fin
* Use name as stas suggestion
* Clean
* docnit
* toctree
* toctree
* Missing ref
* Put in break
* Smaller diff
* Make note on how to use zeroinit
* Make note about accelerator ds plugin
* More docnits
* Apply suggestions from code review
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
* Limit users to not pass in another ds plugin to another accelerator
* not implemented err + Make a note about why no params
* Apply suggestions from code review from Stas
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
* Add deepspeed_plugins arg + update doc
* Plugin -> plugins
* Change enable() -> select()
* Update ref properly + test
* Be consistent, model1,model2...
* first_, second_
* A few more auto values
* Apply suggestions from code review
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>
---------
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>
