pytorch

82154688 - Feature:To add --tolerance option to benchmark scripts (#102218)

Feature:To add --tolerance option to benchmark scripts (#102218) The "tolerance" option evaluates the model on the baseline device in eager mode (default: CPU) compared to the test device (e.g., CUDA, XLA, etc.) and compares the output tensors to determine the absolute tolerance value based on the [formula](https://pytorch.org/docs/stable/generated/torch.allclose.html). It then saves the results in a CSV file. This comparison highlights the tolerance/accuracy difference between XLA and GPU/CPU devices and can also be used to evaluate newer accelerators. This feature aims to identify accuracy failures on the test device (e.g., XLA) and facilitate quick bug triaging. This feature enables the following capabilities: 1. Ability to monitor accuracy issues of backends 2. Provide more informative picture on accuracy beyond pass/ fail status 3. Having a dump of accuracy information will help triage models accordingly The data generated using this feature is in the [spreadsheet](https://docs.google.com/spreadsheets/d/1A8BAzSqfAw0Q5rgzK5Gk__Uy7qhuynh8tedxKnH-t94/edit#gid=0). The spreadsheet data can be used to compile the below summary table: | Suite | Max Tolerance | | No. of models with high inaccuracy(>=0.005) | | Mean Tolerance | | |------------------ |:-------------:|:--------:|:-------------------------------------------:|:--------:|:--------------:|:--------:| | | xla | inductor | xla | inductor | xla | inductor | | huggingface | 0.1169 | 0.0032 | 1 | 0 | 0.0022 | 0.0005 | | timm_models | 0.0373 | 2.8892 | 10 | 8 | 0.0028 | 0.7044 | | torchbench | 3.013 | 3.0381 | 6 | 2 | 0.0016 | 0.0016 | | All models | 3.013 | 3.0381 | 17 | 10 | 0.0028 | 0.7044 | I used PyTorch release/2.0 branch and corresponding [commit_pin](https://github.com/pytorch/pytorch/blob/release/2.0/.github/ci_commit_pins/xla.txt) for XLA to generate the above data. Fixes #ISSUE_NUMBER Pull Request resolved: https://github.com/pytorch/pytorch/pull/102218 Approved by: https://github.com/jansel