Fix infer np scalar dtype mem leak (#24267)
Summary:
Fixes https://github.com/pytorch/pytorch/issues/24200 . I'm a bit worried that the test might be flaky...
Pull Request resolved: https://github.com/pytorch/pytorch/pull/24267
Differential Revision: D17079762
Pulled By: gchanan
fbshipit-source-id: a120688b9583ca4b74bdfb295914298f22540ffd
