Move USE_NUMPY to more appropriate targets (#51143)
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/51143
Test Plan: CI
Reviewed By: wconstab
Differential Revision: D26084123
fbshipit-source-id: af4abe4ef87c1ebe5434938320526a925f5c34c8
