Fix allow_inf in assertEqual (#16959)
Summary:
gchanan pointed out in https://github.com/pytorch/pytorch/pull/16389 that `allow_inf` is treating `-inf` and `inf` as equal. This fixes it.
Also fixing #16448 since it's near and 2.1 has released.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/16959
Differential Revision: D14025297
Pulled By: gchanan
fbshipit-source-id: 95348309492e7ab65aa4d7aabb5a1800de66c5d6
