Reduce set of build/tests which run on PRs. (#20930)
Summary:
Resubmit of #20775
Signed-off-by: Edward Z. Yang <ezyang@fb.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/20930
Differential Revision: D15503173
Pulled By: ezyang
fbshipit-source-id: a5de8eacf6b29ee26f07ac53c915fff3f4d32569
