Revert "Add backward check for test_memory_format (#106104)"
This reverts commit 2e44adb06608d09a36b899ffdb375cb7d46a78d2.
Reverted https://github.com/pytorch/pytorch/pull/106104 on behalf of https://github.com/huydhn due to Sorry for reverting this but it is failing inductor job in trunk https://hud.pytorch.org/pytorch/pytorch/commit/2e44adb06608d09a36b899ffdb375cb7d46a78d2. I will add ciflow/inductor label to the PR make sure that the test runs there ([comment](https://github.com/pytorch/pytorch/pull/106104#issuecomment-1683119990))
