Remove useless test_reference_numerics skip infos (#52890)
Summary:
These are no longer useful. Let's wait for a few days before merging this, just in case somebody finds failures in them.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/52890
Reviewed By: H-Huang
Differential Revision: D26725500
Pulled By: mruberry
fbshipit-source-id: 3ebc18ee11ebef34451e60861414521730742288
