optimize is_float_point calls (#37012)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/37012
Removes an if statement in `torch.nn.functional.affine_grid`
Test Plan: Imported from OSS
Differential Revision: D21160755
Pulled By: eellison
fbshipit-source-id: 8b030936c9fbdb05b44abc9f254805d102f2acc2
