added check for NumberType (#44375)
Summary:
Fixes https://github.com/pytorch/pytorch/issues/44107
Pull Request resolved: https://github.com/pytorch/pytorch/pull/44375
Reviewed By: mrshenli
Differential Revision: D23906728
Pulled By: eellison
fbshipit-source-id: 3b534e5dd3af1f5e43a7314953e64117cbe8ffe4
