Remove downloading the nightly, check only if the workflow is scheduled (#1613)
Summary:
On A10G, we pull the torchbench:latest docker and run the tests in the docker.
On GCP A100 instances, the runner is already in the docker, so we can just skip the pytorch installation.
Test plan of v3 test workflow: https://github.com/pytorch/benchmark/actions/runs/4899087111
Test plan of A100 userbenchmark: https://github.com/pytorch/benchmark/actions/runs/4898890711
Pull Request resolved: https://github.com/pytorch/benchmark/pull/1613
Reviewed By: janeyx99
Differential Revision: D45627169
Pulled By: xuzhao9
fbshipit-source-id: ad166fba0f20a7ed353a5ae47f7fba8c6238a549