ps sparse rpc (#58003)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/58003
adds trainer class DdpTrainer
adds trainer class DdpSparseRpcTrainer
adds server class ParameterServerBase
adds server class AverageParameterServer
adds experiment ddp_cpu_sparse_rpc_nccl_allreduce
adds experiment ddp_cuda_sparse_rpc_nccl_allreduce
quip document https://fb.quip.com/iQUtAeKIxWpF
Test Plan: Imported from OSS
Reviewed By: albanD
Differential Revision: D29379696
Pulled By: gcramer23
fbshipit-source-id: 9cf5fb7398ba2fa3eb694afbddc4ed00d97f205f
