Reduce set of build/tests which run on PRs. (#20775)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/20775
ghimport-source-id: 8d05ed03b8a841233d578a38b7c84bd1152c08e5
Differential Revision: D15499918
Pulled By: ezyang
fbshipit-source-id: 992e3f91f95dd9c0564e5ed6793dd1b286ddba00
