Move libtorch builds to be master-only (#58183)
Summary:
There were almost no libtorch specific regressions recently
Pull Request resolved: https://github.com/pytorch/pytorch/pull/58183
Reviewed By: janeyx99
Differential Revision: D28393091
Pulled By: malfet
fbshipit-source-id: 6dadd915ba574294afa6a95eaa759564af3154d4
