Revert D23490149: [pytorch][PR] Compile less legacy code when BUILD_CAFFE2 is set to False
Test Plan: revert-hammer
Differential Revision:
D23490149 (https://github.com/pytorch/pytorch/commit/15e99b6ff6ee88ef955ee526d35856451536ee99)
Original commit changeset: a76382c30d83
fbshipit-source-id: 75057fa9af2c19eb976962552118bf0a99911b38
