Make some downcast issues explicit (#60412)
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/60412
Test Plan: Sandcastle
Reviewed By: ngimel
Differential Revision: D29243195
fbshipit-source-id: c508b729d6a0e6f8a591521bce788e6cfd8531f8
