Revert "Allow returned tensor impls in fallback (#80545)"
This reverts commit d7e4520d1d4bba1264ca9efec3846045deb18bed.
Reverted https://github.com/pytorch/pytorch/pull/80545 on behalf of https://github.com/malfet due to New test broke rocm, see https://hud.pytorch.org/pytorch/pytorch/commit/d7e4520d1d4bba1264ca9efec3846045deb18bed
