Revert #57979 (and following #58083 #58082) (#58182)
The point of #57979 was to make inference faster, but it made it instead
much slower
(https://github.com/JuliaLang/julia/commit/83524acfee6d1a99d175f0923005a229f9e885e5#commitcomment-155658124),
so revert back to the fast behavior before it was "optimized" (and
revert the bugfixes for the original commit).
