Add regression detector to generate an abtesting config file (#1498)
Summary:
Add a basic regression detector to read in two metrics files and generate their diff.
Pull Request resolved: https://github.com/pytorch/benchmark/pull/1498
Test Plan:
```
$ python regression_detector.py --control ./.userbenchmark/torch-nightly/metrics-20230320145813.json --treatment ./.userbenchmark/torch-nightly/metrics-20230321145813.json
$ cat .userbenchmark/torch-nightly/regression-20230321160820.yaml
control_env:
pytorch_git_version: 4d3baf684b821893b4f6e8915ff55e48a2c8eb34
treatment_env:
pytorch_git_version: 8c8491fe4cc8d0b7d05a4891ce23a1df39b45889
bisection: null
details:
test_train[BERT_pytorch-cuda-eager]_latency:
control: 115.718457
treatment: 115.718457
delta: 0.0
test_train[BERT_pytorch-cuda-eager]_cmem:
control: 2.591796875
treatment: 2.591796875
delta: 0.0
```
Reviewed By: janeyx99
Differential Revision: D44271108
Pulled By: xuzhao9
fbshipit-source-id: 4ab1729f2a5a867d450fa98a7469f201b74e6c06
benchmark
28fb420b - Add regression detector to generate an abtesting config file (#1498)
