Add CI workflow and script to test torchbench. (#56957)
Summary:
This PR adds TorchBench (pytorch/benchmark) CI workflow to pytorch. It tests PRs whose body contains a line staring with "RUN_TORCHBENCH: " followed by a list of torchbench model names. For example, this PR will create a Torchbench job of running pytorch_mobildnet_v3 and yolov3 model.
For security reasons, only the branch on pytorch/pytorch will run. It will not work on forked repositories.
The model names have to match the exact names in pytorch/benchmark/torchbenchmark/models, separated by comma symbol. Only the first line starting with "RUN_TORCHBENCH: " is respected. If nothing is specified after the magic word, no test will run.
Known issues:
1. Build PyTorch from scratch and do not reuse build artifacts from other workflows. This is because GHA migration is still in progress.
2. Currently there is only one worker, so jobs are serialized. We will review the capacity issue after this is deployed.
3. If the user would like to rerun the test, she has to push to the PR. Simply updating the PR body won't work.
4. Only supports environment CUDA 10.2 + python 3.7
RUN_TORCHBENCH: yolov3, pytorch_mobilenet_v3
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56957
Reviewed By: janeyx99
Differential Revision: D28079077
Pulled By: xuzhao9
fbshipit-source-id: e9ea73bdd9f35e650b653009060d477b22174bba
