Fix bug with rebuilding extensions every import (#56015)
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/56015
Reviewed By: mruberry
Differential Revision: D27765934
Pulled By: ezyang
fbshipit-source-id: 65cace951fce5f2284ab91d8bd687ac89a2311fb
