Add the A100 bisection workflow (#2054)
Summary:
We are adding the generic A100 bisection workflow for bisecting any userbenchmark.
The workflow requires 4 arguments:
1. The start pytorch commit hash on the main branch
2. The treatment pytorch commit hash on the main branch
3. The userbenchmark name
4. The userbenchmark arguments to trigger the regression
Pull Request resolved: https://github.com/pytorch/benchmark/pull/2054
Test Plan:
Automatic bisection of the `test_bench` userbenchmark on the accuracy issue of llama_v2_7b_16h:
Start commit hash (2023-11-15): `a5a404865c01f86881f6b3ab0cd9a562d0b420de`
End commit hash (2023-11-16): `690c805c8b539501aad5fbf18914ac92afb65d5a`
Userbenchmark name: `test_bench`
Userbenchmark arguments: `llama_v2_7b_16h -d cuda -t eval --accuracy`
Known root cause commit: `12b2dd16b050e6495910fc564517fbb51dde1f20`
Reviewed By: aaronenyeshi
Differential Revision: D51551127
Pulled By: xuzhao9
fbshipit-source-id: 45385f95a0f30d9eb93efccb32aaa8cf32297d4c
