pytorch

3078233e - [Gradient Compression] Make FP16 compression as a wrapper that can be combined with other communication hooks (#53808)

[Gradient Compression] Make FP16 compression as a wrapper that can be combined with other communication hooks (#53808) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/53808 Create a FP16 wrapper that can combine FP16 gradient compression with any gradient compression algorithm. Test Plan: Unit test: ``` buck test mode/dev-nosan caffe2/test/distributed:c10d -- test_fp16_compress_wrapper ``` Performance Test on DDP QPS Benchmark: Check if AllReduce + FP16 Wrapper = FP16 Compression 1) FP16 Compression: f256897690 2) FP16 Wrapper + AllReduce (after patching D26960986): f256897289 Reviewed By: SciPioneer Differential Revision: D26978832 fbshipit-source-id: 0dcd18b050c02f5e9f3cff56344d1f39a04e20c0