inference: stop re-converging worlds after optimization (#38820)
The validity did not change, so we should not need to update it. This
also ensures we copy over all result information earlier, so we can
destroy the InferenceState slightly sooner, and slightly cleaner data flow.
(cherry picked from commit 8c014445a7e6ba515286e38fe4585510cc93d300)
