Improve the correctness checking code and fix non-deterministic models (#997)
Summary:
`pyhpc_turbulent_kinetic_energy` model is non-deterministic because its code modifies the input tensor and returns it at the same time. Therefore, if users run `eval()` on the same model instance multiple times, they get different results. (https://github.com/dionhaefner/pyhpc-benchmarks/blob/master/benchmarks/turbulent_kinetic_energy/tke_pytorch.py#L258)
This PR copies the data before running the `integrate_tke` function, fixing this problem.
We also improved the correctness checking code to reuse the code in torchdynamo https://github.com/pytorch/torchdynamo/blob/main/torchdynamo/utils.py#L411, and do a compulsory check for correctness across all devices on inference, so that downstream doesn't need to workaround it.
Fixes https://github.com/pytorch/benchmark/issues/891
Pull Request resolved: https://github.com/pytorch/benchmark/pull/997
Reviewed By: erichan1
Differential Revision: D37497341
Pulled By: xuzhao9
fbshipit-source-id: 3d53a390ec4c3b5f6114891386072917f8cb6142