Don't build deps for `python setup.py egg_info` (#36208)
Summary:
Fixes https://github.com/pytorch/pytorch/issues/36207.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/36208
Differential Revision: D20919649
Pulled By: ezyang
fbshipit-source-id: b5242a540181b29dba8987fb5f00332e1e81ca98
