Fix incorrect DDP test (#64074)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/64074
Previous PR https://github.com/pytorch/pytorch/pull/63831 did not actually test the error in https://github.com/pytorch/pytorch/issues/63812. Introduce a test
directly from the repro that simulates it.
ghstack-source-id: 137171460
Test Plan: CI
Reviewed By: SciPioneer
Differential Revision: D30569719
fbshipit-source-id: fd61250ef6d291c093607663d91d6d2cb5574eb7
