[Constraint graph] Use adjacency info for constraint gathering.
This reinstates the use of direct adjacency information when gathering
constraints, effectively reverting
54bdd7b840721523f363e343d7b25997a8a332fe.
Fixes the regression that commit caused, which is tracked by
rdar://problem/54274245.
