Add OpInfo for polygamma and remove torch_op_tests Infra (#51966)
Summary:
Reference: https://github.com/pytorch/pytorch/issues/42515
* OpInfo entry for Polygamma
* Removes infra of torch_op_tests
Pull Request resolved: https://github.com/pytorch/pytorch/pull/51966
Reviewed By: bdhirsh
Differential Revision: D27851858
Pulled By: mruberry
fbshipit-source-id: 7f1d0273065e1df56a152f95a14513959af29a1b
