Don't clobber pytorch image with libtorch build. (#28581)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/28581
Fixes #28305
Signed-off-by: Edward Z. Yang <ezyang@fb.com>
Test Plan: Imported from OSS
Differential Revision: D18124450
Pulled By: ezyang
fbshipit-source-id: 0d4bb99a6bdff9ddbfb4d25cc0f67cc261ed26ba
