use memory locations instead of values for working set (#21223)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/21223
ghimport-source-id: 82800a465a4273e185bfffe2f67835b2f7f3a519
Reviewed By: jamesr66a
Differential Revision: D15592317
Pulled By: suo
fbshipit-source-id: 5e87c803a928b61c923200888a3ff1ac7b2523e0
