[ty] Compact retained semantic maps (#25238)
## Summary
This is a follow-up to #25102 to apply the same principles to a few more
collections. The benchmarks look good (-1% on some micro-benchmarks, +1%
on some real projects) and there are moderate memory savings.
