cleanup remset logic a bit (#55021)
I think that keeping a single `remset` (instead of two and keep
alternating between them) should be a bit easier to understand and
possibly even a bit faster (since we will be accessing the `remset` only
once), though that should be a very small difference.
