Cleanup the operators directory (#2539)
Summary:
We still have a leftover operators directory due to ShipIt issues with code sync. This PR will clean this up.
Pull Request resolved: https://github.com/pytorch/benchmark/pull/2539
Reviewed By: danzimm
Differential Revision: D65945850
Pulled By: xuzhao9
fbshipit-source-id: 00c7f8de06ca35fd5a978ad0f4f04b3cba4b5561
