[FX] Fix handling of `inf` and `nan` literals (#46894)
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/46894
Test Plan: Imported from OSS
Reviewed By: zdevito
Differential Revision: D24555136
Pulled By: jamesr66a
fbshipit-source-id: 22765a4d9d373711e9e6d7b1d3898080ecbcf2f5
