Cleanup unnecessary SpectralFuncInfo logic (#48712)
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/48712
Test Plan: Imported from OSS
Reviewed By: ngimel
Differential Revision: D25868675
Pulled By: mruberry
fbshipit-source-id: 90b32b27d9a3d79c3754c4a1c0747dbe0f140192
