Verify indirect deps reachable on incremental run (#20735)
Fixes https://github.com/python/mypy/issues/19477
This is another surprising correctness hole in the indirect dependencies
logic. We implicitly assume everywhere that indirect dependencies are
subset of transitive ones. Although it is true when the indirect
dependencies are initially calculated, there is no guarantee this will
stay true after an incremental update where some import structure has
changed.
Currently, we only handle the situations where an indirect dependency is
removed from the graph completely, but if it is still in the graph, but
now brought in by a _different_ dependent, this may cause out-of-order
cache loading, and thus a bug or a crash. It is interesting that
although in theory it is kind of a big deal, in practice it is very hard
to create such scenario. So far I found only two complicated scenarios
(see tests).
Implementation note: although this additional check is needed for rare
edge cases, it is quite computationally expensive. I tried few options
(including only computing import deltas), but the only viable/robust
option seems to be an honest recursive check with some caching. I tested
this on `torch` (it has ~1000 small SCCs plus single SCC with around
1000 modules), and the performance penalty is negligible there. So
hopefully it should be fine even for large code bases.
