Revert "make sure all torch._numpy tests run on CI (#108762)"
This reverts commit 7abeb92796635bd3ee216a0872bddd0395e97d10.
Reverted https://github.com/pytorch/pytorch/pull/108762 on behalf of https://github.com/clee2000 due to sorry but I think the asan test_scalarmath failure is real https://hud.pytorch.org/pytorch/pytorch/commit/7abeb92796635bd3ee216a0872bddd0395e97d10 https://github.com/pytorch/pytorch/actions/runs/6132913963/job/16645381921 ([comment](https://github.com/pytorch/pytorch/pull/108762#issuecomment-1714214523))
