Revert "Added conv constraint that infers layouts (#89031)" (#89451)
This reverts commit 716f70f19a4b63268da2a753afdbe9b385a831ab.
Fixes performance regression and compilation latency increase.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/89451
Approved by: https://github.com/soumith, https://github.com/jansel
