[caffe2] Revert D22220798 (#41302)
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/41302
Test Plan:
```
buck test //caffe2/caffe2/fb/predictor:black_box_predictor_test
```
Differential Revision: D22492356
fbshipit-source-id: efcbc3c67abda5cb9da47e633804a4800d92f89b
