Revert D28875276: Move RPC agents to libtorch
Test Plan: revert-hammer
Differential Revision:
D28875276 (https://github.com/pytorch/pytorch/commit/fc50f91929b67f1ccf34890416b46a17b399e21b)
Original commit changeset: f2f6970fd74d
fbshipit-source-id: 3c52af652579733ebea8ddfb06576a0ce262bf78
