Unit tests for MiCS (#4792)
In response to the ask from
https://github.com/microsoft/DeepSpeed/pull/2964#issuecomment-1832161865,
I added three more unit tests related to MiCS.
There are two knowledge issues:
- Testing on Torch 2.1.0 triggers `_IllegalWorker` in coalesced all
gather. I made changes to ignore this condition. and Currently, I don't
know the reason.
- The MiCS implementation is not working with offloading, so the failure
in `TestZeroPartialOffloadConfigSweep` is expected.
---------
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
