Fix a downcast (#47919)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/47919
Suppresses a downcast warning.
Test Plan:
Reproduces with
```
buck test mode/dev-nosan //caffe2/torch/fb/sparsenn:gpu_test
```
Reviewed By: suphoff
Differential Revision: D24866987
fbshipit-source-id: 44f19ab37a7d95abe08f570abfebc702827a2510
