DeepSpeed
39a682d7 - Low-precision master params/grads/optimizer states (#7700)

Commit
353 days ago
Low-precision master params/grads/optimizer states (#7700) DeepSpeed optimizer always creates fp32 master params/gradients/optimizer states. However, we sometimes want to keep them lower precision given [torch.autocast support](https://deepspeed.readthedocs.io/en/latest/training.html#mixed-precision-training). This PR allows lower precision master params/grads/optimizer states with bf16/fp16 enabled. DeepSpeed currently accepts `fp16_master_weights_and_gradients` option under `fp16` section (not documented) with ZeRO1/2. This PR extends this for bf16 and also ZeRO3. In `bf16` section, we can have new items `bf16_master_weights_and_grads` and `bf16_optimizer_states`. Similary to `fp16_master_weights_and_grads`, `bf16_master_weights_and_grads` keeps master parameters in bf16. `bf16_optimizer_states` keeps optimizer states also in bf16. Here is an example configuration: ```json "bf16": { "enabled": true, "bf16_master_weights_and_grads": true, "bf16_optimizer_states": true } ``` Note that `bf16_master_weights_and_grads==True` and `bf16_optimizer_states==False` is supported only with cpu offloading. Also, we don't have `fp16_optimizer_states` as it won't be practical. More details are described in [`config-json.md`](https://github.com/tohtana/DeepSpeed/blob/88e0bbdfba89c4712d815980ddb28353d6da5b2e/docs/_pages/config-json.md) Previously, `torch.autocast` support (`torch_autocast` section in config) was not compatible with `bf16` `fp16` enabled, but we now accept the combination. This PR also adds some test cases for the configurations as well as the combination with `torch.autocast`. --------- Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com> Co-authored-by: Olatunji Ruwase <tunji.ruwase@snowflake.com>
Author
Parents
Loading