[Inductor Perf Test Workflow] Remove pull request trigger and rely on ciflow/ label only (#95755)
Mitigates A100 queue issue.
Workflow seems to run twice upon pull request changes.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/95755
Approved by: https://github.com/seemethere
