Fix CPU-only binary testing by properly installing cpu-only first. (#23611)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/23611
Signed-off-by: Edward Z. Yang <ezyang@fb.com>
Test Plan: Imported from OSS
Differential Revision: D16601098
Pulled By: ezyang
fbshipit-source-id: febb5a822854b91d5b3d942e6bf71b4ae9f1f15c
