Compile less legacy code when BUILD_CAFFE2 is set to False (#44079)
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/44079
Reviewed By: walterddr
Differential Revision: D23490149
Pulled By: malfet
fbshipit-source-id: a76382c30d83127d180ec63ac15093a7297aae53
