[Model Averaging] Post-localSGD optimizer (#62131)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/62131
Wrap `PeriodicModelAverager` as an optimizer.
Currently both the optimizer and averager require an input `params` arg, where the latter actually can read params from the optimizer wrapper. Will update averager class API in a follow-up PR.
Proposal: https://github.com/pytorch/pytorch/issues/59699
ghstack-source-id: 134560248
Test Plan: buck test mode/dev-nosan caffe2/test/distributed:distributed_nccl_fork -- test_post_localSGD_optimizer_parity
Reviewed By: rohan-varma
Differential Revision: D29881465
fbshipit-source-id: b9634972f4d8bffd3b3eb94f5dbbb19db2bcd759
