Save a little bit of work in constant pooling by not moving nodes that will get deleted.
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/16161
Differential Revision: D13791247
Pulled By: resistor
fbshipit-source-id: 2a5a4f98309509b4ba875373ee57e6f63c75a4fd
