Don't run_build when run_binary_tests (#59982)
Summary:
https://github.com/pytorch/pytorch/issues/59889 wasn't a proper revert of https://github.com/pytorch/pytorch/issues/58778. This PR fixes that.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/59982
Reviewed By: seemethere
Differential Revision: D29114129
Pulled By: samestep
fbshipit-source-id: b40563db6ff1153a5f759639978279f5fcbccaa9
