Fix the offset numerical bug when casting (#21484)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/21484
cast<int32_t*> => cast<int32_t>
Also fixed reserve problem which might cause incorrect pointer.
Reviewed By: yinghai
Differential Revision: D15699866
fbshipit-source-id: 374418476bddd60f5c5306c8c57319ccf28b9990
