Remove ignored modes workaround (#135502)
Summary:
X-link: https://github.com/pytorch/pytorch/pull/135502
Approved by: https://github.com/anijain2305
ghstack dependencies: #134732, #133137, #135443, #135444, #135422
Reviewed By: jeanschmidt
Differential Revision: D62737308
Pulled By: mlazos
fbshipit-source-id: d436c5b5db309e64f17d51dad30a8936567f9a3e
