[mypyc] Enable incremental self-compilation (#21369)
Six fixes on top of #21299, all required to self-compile mypy or to
install a `separate=True` wheel via pip.
- `mypyc/build.py`: pip invokes `setup.py` twice when building a wheel.
On the second invocation mypy's incremental cache is fully warm, so we
generate no new C source for any group; the resulting extensions ship
without their entry points and import as stubs.
- **Fix**: when a group emits no C source, reuse the .c file from the
previous pass.
- `mypyc/codegen/{emit,emitfunc}.py`: when code in one compiled group
reads an attribute on an object whose class lives in another group, the
generated cast depends on that other group's struct definitions. We
weren't recording the dependency, so the C compiler couldn't see the
layout and the build failed.
- **Fix**: register the dependency at the cast site.
- `mypyc/codegen/emitmodule.py`: when mypy compiles itself, a generated
shim file can share a basename with a runtime C file. The C compiler
resolves the runtime include relative to the shim's directory and picks
up the shim instead.
- **Fix**: search the include path explicitly so shims can't shadow
runtime files.
- `mypyc/lib-rt/misc_ops.c`: each compiled module gets its own shared
library next to it in the package tree. The runtime was computing the
module's file path as if a single shared library sat above the whole
package, which doubled the package prefix and broke submodule lookups.
- **Fix**: detect the per-module case and use only the module's leaf
name.
- `mypyc/irbuild/prepare.py`: traits and builtin-derived classes don't
get a real C constructor emitted. A clean build sidesteps that, but a
fully cached rebuild was taking the direct-call path and producing C
that referenced a constructor that doesn't exist.
- **Fix**: skip the registration the same way a clean build does.
- `mypyc/build.py`: on every build_ext, setuptools rewrites every
compiled .so in the source tree even when nothing changed. On macOS this
invalidates the OS signature cache, so every import on the next run pays
a re-verification cost.
- **Fix**: skip the copy when source and destination already match,
taking a 1-line edit rebuild from ~72s to ~6s. This is really a `setup`
tools limitation though (relevant [mypy
issue](https://github.com/mypyc/mypyc/issues/1068) ?)
I also added a `MYPYC_SEPARATE` env knob so CI can exercise the codegen
path against mypy itself.
## Benchmarks
Mypy self-compile on macOS, `MYPYC_OPT_LEVEL=0`, `-j 11`. Three
scenarios:
| | monolithic | separate=True |
|:---:|:---:|:---:|
| Clean build | 180s | 108s |
| No-op rebuild | 124s | 5s |
| 1-line edit | 106s | 6s |
