ruff
8656c731 - [ty] Compact indexed AST node storage (#25998)

Commit
41 days ago
[ty] Compact indexed AST node storage (#25998) ## Summary `IndexedModule` retains one lookup entry for every indexed AST node. Prior to this change, each entry was a full `AnyRootNodeRef`, even though resolving a `NodeIndex` only requires the node's address and root AST kind. An earlier version of this PR chose one packed, split, or wide representation for the entire module. Corpus data showed that the split layout was still used by 23.2% of modules, while the module-wide base also meant that one distant allocation could widen the representation for the entire AST. We explored a chunked replacement in #26047 and have now folded that implementation back into this PR. The final representation divides the index into fixed groups of 64 nodes. Each chunk chooses its own base address and the minimum entry width needed for its nodes. Aligned relative offsets and five-bit root-node kinds share a packed `u64` bitstream, while unusually large or unaligned chunks use a local wide fallback. Deriving the chunk directly from the node index keeps lookup constant-time, prevents unrelated AST allocations from affecting one another, and removes the previous split representation. The review cycle also tightened the raw-pointer boundary. `AnyRootNodeRef` now decomposes to `NonNull<()>`, its unsafe reconstruction API documents the complete caller contract, and `IndexedNodes` records the address-stability, provenance, liveness, and node-kind invariants used during lookup. Collection retains typed `AnyRootNodeRef` values until encoding, and the coverage round-trips real indexed AST pointers rather than fabricated addresses. The AST generator emits `RootNodeKind` and the raw-parts conversions alongside `AnyRootNodeRef`, keeping the unflattened root-type mapping synchronized as the AST evolves. This is intentionally separate from `NodeKind`, which distinguishes concrete variants such as `StmtFunctionDef` rather than the `Stmt` root type addressed by the index. The traversal also stops indexing annotation expressions twice, since `walk_annotation` already delegates to `visit_expr`. The latest memory report for the final chunked implementation shows a 14.7–14.9% reduction in retained `parsed_module` memory across the measured projects, reducing overall memory by 0.6–5.1%.
Author
Parents
Loading