Low-precision master params/grads/optimizer states (#7700)
DeepSpeed optimizer always creates fp32 master
params/gradients/optimizer states.
However, we sometimes want to keep them lower precision given
[torch.autocast
support](https://deepspeed.readthedocs.io/en/latest/training.html#mixed-precision-training).
This PR allows lower precision master params/grads/optimizer states with
bf16/fp16 enabled.
DeepSpeed currently accepts `fp16_master_weights_and_gradients` option
under `fp16` section (not documented) with ZeRO1/2. This PR extends this
for bf16 and also ZeRO3.
In `bf16` section, we can have new items `bf16_master_weights_and_grads`
and `bf16_optimizer_states`.
Similary to `fp16_master_weights_and_grads`,
`bf16_master_weights_and_grads` keeps master parameters in bf16.
`bf16_optimizer_states` keeps optimizer states also in bf16. Here is an
example configuration:
```json
"bf16": {
"enabled": true,
"bf16_master_weights_and_grads": true,
"bf16_optimizer_states": true
}
```
Note that `bf16_master_weights_and_grads==True` and
`bf16_optimizer_states==False` is supported only with cpu offloading.
Also, we don't have `fp16_optimizer_states` as it won't be practical.
More details are described in
[`config-json.md`](https://github.com/tohtana/DeepSpeed/blob/88e0bbdfba89c4712d815980ddb28353d6da5b2e/docs/_pages/config-json.md)
Previously, `torch.autocast` support (`torch_autocast` section in
config) was not compatible with `bf16` `fp16` enabled, but we now accept
the combination.
This PR also adds some test cases for the configurations as well as the
combination with `torch.autocast`.
---------
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Co-authored-by: Olatunji Ruwase <tunji.ruwase@snowflake.com>