Fix macos public bindings failures (#80970)
We are seeing unrelated public bindings test failures on macos tests being triggered on random PRs. Here's an attempt to fix some of those.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80970
Approved by: https://github.com/rohan-varma
