Revert D28143091: [pytorch][PR] Add cross OpInfo
Test Plan: revert-hammer
Differential Revision:
D28143091 (https://github.com/pytorch/pytorch/commit/4a872f85396b426f7e26ee20c25e7916d45f70e2)
Original commit changeset: 0b98226a1811
fbshipit-source-id: eda38923f31ac5a79af5c78077ed0106d904f6da
