Remove more error-exposing tests in exp that cannot be reliably reproduced (#40825)
Summary:
Continuing https://github.com/pytorch/pytorch/issues/40824
All CIs have been enabled (on a branch that starts with `ci-all/`)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/40825
Differential Revision: D22328732
Pulled By: ezyang
fbshipit-source-id: 3e517d01a9183d95df0687b328fb268947ea5fb0
