fix accuracy failure for a few models under freezing (#121054)
Summary:
Fix https://github.com/pytorch/pytorch/issues/120545 . The reason why these models fail accuracy test with freezing is due to the conv-batchnorm fusion. Conv-batchnorm fusion causes relative big numerical churn.
For the failed TIMM models, raising the tolerance to `8 * 1e-2` can make the test pass.
For the failed TB models, the numerical difference is too large. Having a discussion with eellison , we decided to skip them with freezing for now.
One the other hand, we probably should dig more why the conv-bn fusion cause such large numerical difference.
X-link: https://github.com/pytorch/pytorch/pull/121054
Approved by: https://github.com/eellison
Reviewed By: izaitsevfb
Differential Revision: D54472101
Pulled By: shunting314
fbshipit-source-id: ae73f623153d0df647a333b6d98bfe6a879b359c