Revert "Build and run Metal tests in CI (#86062)"
This reverts commit f88bf8de2cba75377baf469b3dd3f8bc415ee7d2.
Reverted https://github.com/pytorch/pytorch/pull/86062 on behalf of https://github.com/huydhn due to Breaking trunk https://hud.pytorch.org/pytorch/pytorch/commit/f88bf8de2cba75377baf469b3dd3f8bc415ee7d2
