Compile less legacy code when BUILD_CAFFE2 is set to False (take 2) (#44453)
Summary:
2nd attempt to land https://github.com/pytorch/pytorch/pull/44079
Pull Request resolved: https://github.com/pytorch/pytorch/pull/44453
Reviewed By: walterddr, seemethere
Differential Revision: D23619528
Pulled By: malfet
fbshipit-source-id: c7c206ebd327dcf3994789bd47008b05ff862fe7
