Revert D25093315: [pytorch][PR] Fix inf norm grad
Test Plan: revert-hammer
Differential Revision:
D25093315 (https://github.com/pytorch/pytorch/commit/ca880d77b81f5318d3941266a870a982d04b1db5)
Original commit changeset: be1a7af32fe8
fbshipit-source-id: b383ec2a2c5884149b4fc7896f9d2856259794cd
