next.js

ebcc2840 - perf(turbopack): canonicalize async loader availability (#100705)

perf(turbopack): canonicalize async loader availability (#100705) ### What? Canonicalize async-loader availability before constructing loaders for dynamic `import()` and `next/dynamic`, so parents that make the same modules available share loader tasks instead of creating them once per page. Lazy-compilation proxies keep their activation paths stable; their actual targets still use canonicalization when their closures contain no further lazy proxies. ### Why? `AsyncLoaderModule` and `async_loader_chunk_item` are keyed by `AvailabilityInfo`. Both the available chunk groups and `entry_group` carry per-parent information. On v0 there are about 196K `async_loader_chunk_item` tasks for a few hundred targets. Intersecting availability with reachable groups alone isn't enough: a module shared with every page, such as `react/jsx-runtime`, keeps that intersection different per page. ### How? - **Precompute observed sets in `compute_chunk_group_info`.** Build a chunk-group graph from async, shared/shared-merged and collected references, then close over it per strongly connected component. For each async target, record the distinct chunk-group sets tested during chunking, their union, and whether it needs `entry_group`. Store this in a keyed cell on `ChunkGroupInfo`. - **Canonical availability.** From the union, remove each observed set that misses the parent availability. The result answers every observed intersection question like the original and depends only on those answers, not the parent. It can include groups not originally available; it is a representative, not a subset. Drop `entry_group` when it cannot be read. - **Filter inline at both call sites.** `AvailabilityInfo::canonicalize_for_async_loader` is used in `make_chunk_group` and next-api's `next/dynamic` loader construction. No filtering task per parent/target; equal resulting bitmaps share a cell. Unknown collected subgraphs conservatively keep the original availability. - **Lazy-compilation boundary.** `LazyCompilationProxyModule` implements the generic `ActivatableModule` marker. The pass stops at its async group and omits that group's narrowing entry, preserving the proxy loader's availability. Deferred references must start independent chunk groups, as documented by the marker. - Async ancestors reaching a proxy use the subset-safe intersection filter and keep `entry_group`, including for collecting modules that appear only on activation. - This prevents invented availability reaching an unknown target and keeps nested proxy stub paths stable when activation expands the graph. - Actual targets with proxy-free closures still canonicalize. Ancestors containing more lazy proxies may remain per-parent; this does not promise full deduplication in lazy mode. - **Distinct manifest names.** Base `ManifestAsyncModule::content_ident` on the manifest module's own ident, not the inner module's ident. Empty availability no longer makes the manifest and inner module collide; the lazy activation key remains part of the path. - **Cached chunk-group identity hashes.** `ChunkGroupInfo::chunk_group_ident_hashes` renders and hashes each chunk-group identity once per graph. `AvailableChunkGroups::hash` then reads that one cell, sorts its members' identity hashes and combines them, instead of rendering every member identity in every availability hash. The second commit fixes two reproducible lazy-import regressions exposed by the initial canonicalization: duplicate output paths and activation 404s. They were not flakes. The third commit addresses the vercel-site cold-build CPU regression found in benchmarks: canonical availability sets were larger, and `AvailableChunkGroups::hash` repeated identity rendering for every member. Output names intentionally change because the set hash now combines per-identity hashes. ### Measurement Measured before rebasing onto canary's `Bitset` migration (#100643, #100630); not re-measured since. Production build of a generated app: N client pages, each using `next/dynamic` on the same five targets with private dependencies; JSX targets additionally share `react/jsx-runtime` with every page. Lazy compilation is off. | Variant | N | `async_loader_chunk_item` tasks | `AsyncLoaderModule::new` tasks | Total tasks | |---|---|---|---|---| | plain | 10 | 100 → 10 | 150 → 15 | 115,052 → 112,153 | | plain | 100 | 1,000 → 10 | 1,500 → 15 | 237,992 → 206,203 | | JSX | 10 | 100 → 10 | 150 → 15 | 115,131 → 112,257 | | JSX | 100 | 1,000 → 10 | 1,500 → 15 | 238,071 → 206,307 | `compute chunk group info` CPU: 2.81 → 3.64 ms at N=100, including 0.75 ms for the new pass. After the lazy fix, quick N=100 runs of both variants still produce 10 loader tasks and 15 loader-module tasks. **No established build-wall-time gain:** the small synthetic app's timings overlap. Not measured on v0. After the third commit, on the JSX app at N=100, availability-hash task executions stay at 5, the new graph task runs once, and `AssetIdent::to_string` calls fall from 13,014 to 11,247 in both runs. One trace per arm reduces total `AvailableChunkGroups::hash` span CPU from 3.10 ms to 1.72 ms, including the first identity-hash computation. Real-app benchmarks have not been rerun after this commit. ### Verification - Unit coverage: disjoint and shared targets; nested async/shared/merged references and cycles; isolated/traced references; collect/emit and entry-group decisions; keyed-cell invalidation; exhaustive canonical and subset-filter properties. - Activation regression test compares inactive/active graphs by chunk-group keys, checks stable subset availability and retained `entry_group` above a proxy, unchanged proxy availability, and canonicalization of activated and unrelated targets. - New `collect/async` execution fixture. - Lazy dev suites: `lazy-dynamic-imports` (19/19), `lazy-dynamic-chunk-list-collision` (1/1), and collision again with cache components and cached navigations (1/1). - Focused Turbopack dev and production suites: `layout-segment-chunk-sharing`, `app-dir/dynamic`, `app-dir/dynamic-import`, `404-page-router`, `next-dynamic`, `pages-app-next-dynamic-preload`. - All 458 Rust snapshot/execution tests pass. Initial optimization renamed 45 outputs with identical mapped contents; the lazy fix introduces no additional snapshot changes. The identity-hash commit renames 15 more outputs; a mechanical check maps all 15 and finds identical contents apart from the changed names. - Core/ecmascript units: 813 passed, 2 skipped. Workspace fmt, check, clippy and docs pass with warnings denied. After the identity-hash change, lazy imports (19/19), lazy collision (1/1), and `app-dir/dynamic` production (12/12) pass with a rebuilt native binary. ### Out of scope - Lowering entry groups into chunking edges or narrowing them to subsets. - Canonicalizing proxy loaders themselves or resolving their manifests by activation key on the server. <!-- NEXT_JS_LLM_PR --> <!-- fleet 4f67c6f7-bbe2-454b-9a17-822a60ef8f7e --> --------- Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com> Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>