Add DDP Communication Hooks (#2841)
* Add ddp comm hook
* Fix dataclass order
* Merge ddp grad hook to ddp kwargs handler
* Reset ddp kwargs key
* Add test
* Fix test case
* Split ddp grad test
* Fix test case
* Ehance docstring
* Minor
* Use naive baseenum for ddp comm hook type
* Add by feature example
* Add multi device deco
* Add user guide
* Update examples/by_feature/ddp_comm_hook.py
Co-authored-by: Zach Mueller <muellerzr@gmail.com>
* Update examples/by_feature/ddp_comm_hook.py
Co-authored-by: Zach Mueller <muellerzr@gmail.com>
* Add wrapper and state option details
* Update toctree
* Update docs/source/usage_guides/ddp_comm_hook.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/usage_guides/ddp_comm_hook.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/usage_guides/ddp_comm_hook.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/usage_guides/ddp_comm_hook.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/usage_guides/ddp_comm_hook.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/usage_guides/ddp_comm_hook.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/usage_guides/ddp_comm_hook.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Mv ddp comm hook index
* Fix ddp comm hook user guid
* Del empty line
---------
Co-authored-by: Zach Mueller <muellerzr@gmail.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
