Remove some unnecessary cast for complex numbers. (#38422)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/38422
This partially reverts #38021, due to the availability of #38418
Test Plan: Imported from OSS
Differential Revision: D21587201
Pulled By: malfet
fbshipit-source-id: c0717303c842ceb3a202986ec0e808ed45f682f1
