Preserve some lost narrowing, cleanup (#20674)
#20660 is proving a little tricky, so I'm going to try and land some of
the diff independently.
I think of this as mostly a cleanup PR, but it does change a few
semantics. In particular, the places where we previously added a `{}` to
the list of type maps used to clobber some narrowing
