Add tests for DDP control flow models. (#47206)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/47206
As discussed offline with pritamdamania87, add testing to ensure per-iteration and rank-dependent control flow works as expected in DDP with `find_unused_parameters=True`.
ghstack-source-id: 115854944
Test Plan: CI
Reviewed By: pritamdamania87
Differential Revision: D24659901
fbshipit-source-id: 17fc2b3ebba9cef2dd01d2877bad5702174b9767
