Cleans up type conversions, adds CPU test comparing with NumPy (#35374)
Summary:
Per title. Follow-up to https://github.com/pytorch/pytorch/pull/35086.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/35374
Differential Revision: D20712443
Pulled By: mruberry
fbshipit-source-id: 987089c14bff644fd6a636da5530dc260e1d1a68
