benchmark
1e96f5d3 - Add accuracy metrics to e2e models (#1356)

Commit
3 years ago
Add accuracy metrics to e2e models (#1356) Summary: This PR makes two changes to E2E models: 1. Turn on accuracy validation in train by default 2. Store the accuracy metrics in the result json file For example: ``` $ python run_e2e.py hf_bert -t train {"device": "cuda", "device_num": 1, "test": "train", "num_examples": 8576, "num_epochs": 3, "batch_size": 32, "result": {"latency": 8.417137417, "qps": 3056.6211201491665, "accuracy": 0.4213272367274183}} ``` ``` $ python run_e2e.py hf_t5 -t train {"device": "cuda", "device_num": 1, "test": "train", "num_examples": 610336, "num_epochs": 3, "batch_size": 32, "result": {"latency": 141.87018556799998, "qps": 12906.221223784734, "accuracy": 27.662052905235683}} ``` Pull Request resolved: https://github.com/pytorch/benchmark/pull/1356 Reviewed By: yanboliang Differential Revision: D42348124 Pulled By: xuzhao9 fbshipit-source-id: 9064751bb6cbd01d133c12bcbfd142b255ac078b
Author
Parents
Loading