Run test_spectral_ops serially to fix CUDA illegal memory access (#91264)
Fixes https://github.com/pytorch/pytorch/issues/88916
* Running this test sequentially is not flaky after 1000 reruns `pytest --verbose test_spectral_ops.py -k test_fft_round_trip_cuda_float32 --flake-finder --flake-runs=1000`
* On the other hand, the curious thing is that when I run this same command on an active runner with some testing processs running in the background, the reruns could fail with CUDA illegal memory access error (hard to reproduce though) https://paste.sh/6sZdRn95#pve73riXC5XehCLqxlCbnjea. This points to the fact that running the `test_spectral_ops` test in parallel with others might be the surface-level cause of flakiness
So this PR adds the test to the serial list instead. This shouldn't cause any issue w.r.t TTS because the test takes only half a minute at most to finish.
```
+---------------------+-------------------------------------------------+-------------+---------------------+
| file | base_name | test_config | time |
+---------------------+-------------------------------------------------+-------------+---------------------+
| "test_spectral_ops" | "cuda11.6-py3.10-gcc7-sm86" | "default" | 5.991666666666661 |
| "test_spectral_ops" | "cuda11.6-py3.10-gcc7-sm86" | "slow" | 0.18433333333333346 |
| "test_spectral_ops" | "linux-bionic-cuda11.6-py3-gcc7-slow-gradcheck" | "default" | 9.866000000000003 |
| "test_spectral_ops" | "linux-bionic-cuda11.6-py3.10-gcc7" | "default" | 10.591333333333337 |
| "test_spectral_ops" | "linux-bionic-cuda11.6-py3.7-gcc7-debug" | "default" | 11.395000000000003 |
| "test_spectral_ops" | "linux-bionic-cuda11.7-py3.10-gcc7" | "default" | 9.424 |
| "test_spectral_ops" | "linux-bionic-cuda11.7-py3.7-gcc7-debug" | "default" | 8.889000000000003 |
| "test_spectral_ops" | "linux-bionic-py3.7-clang9" | "crossref" | 6.280333333333329 |
| "test_spectral_ops" | "linux-bionic-py3.7-clang9" | "default" | 12.182999999999998 |
| "test_spectral_ops" | "linux-bionic-py3.7-clang9" | "dynamo" | 11.124999999999984 |
| "test_spectral_ops" | "linux-bionic-py3.7-clang9-slow" | "slow" | 0.1916666666666668 |
| "test_spectral_ops" | "linux-focal-py3.7-clang7-asan" | "default" | 20.899666666666658 |
| "test_spectral_ops" | "linux-focal-py3.7-gcc7" | "default" | 5.097999999999996 |
| "test_spectral_ops" | "linux-focal-rocm5.3-py3.8-slow" | "slow" | 0.23700000000000018 |
| "test_spectral_ops" | "macos-12-py3-arm64" | "default" | 2.8396666666666626 |
| "test_spectral_ops" | "macos-12-py3-x86-64" | "default" | 8.838999999999997 |
| "test_spectral_ops" | "parallelnative-linux-focal-py3.7-gcc7" | "default" | 5.016999999999998 |
| "test_spectral_ops" | "win-vs2019-cpu-py3" | "default" | 8.351666666666665 |
| "test_spectral_ops" | "win-vs2019-cuda11.6-py3" | "default" | 27.121666666666687 |
| "test_spectral_ops" | "win-vs2019-cuda11.7-py3" | "default" | 24.567000000000025 |
+---------------------+-------------------------------------------------+-------------+---------------------+
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/91264
Approved by: https://github.com/clee2000