Revert "Re-land: Fix model initialization. (#6182)" (#6220)
This reverts commit 38e364432c77e27f3a2e8f1deaa43faa91e5d812 i.e. #6182. It is causing regressions for Inductor on inference -- see https://github.com/pytorch/xla/pull/6182#discussion_r1430373949
