[caffe2/libtorch] Remove already-owned source
Summary:
This source is already owned by a more fine-grained rule, so avoid a
package boundary violation by having it also be owned by an outer
rule.
Test Plan: CI
Reviewed By: aniketmathur
Differential Revision: D29422794
fbshipit-source-id: 432accc969abcb4d56bd97341a07029926939ea0
