Turbopack: track availability by chunk group (#98685)
### What?
Replaces Turbopack's retained available-module sets with compact
chunk-group bitmaps.
This is PR 5 of a six-PR stack, based on `canary` (#98733, #99686 and
#99749 are already merged):
1. #98823 — Pages Router dev client graph chaining
2. #99910 — regression test for Pages Router `_app` availability
(#99789)
3. #98734 — use graph-derived merged groups
4. #99750 — chunking takes a `ChunkGroupKey`
5. **This PR** — chunk-group availability bitmaps
6. #99050 — keyed chunk-group index lookups
### Why?
`AvailabilityInfo` previously retained every concrete module, module
batch, and async-loader item emitted by ancestor chunk groups. The same
module lists were duplicated across availability chains, increasing
memory use for large graphs.
The module graph already records which chunk groups contain each module.
Retaining only available group indices lets availability checks reuse
that information.
### How?
- Stores available chunk-group indices in a roaring bitmap
(`AvailableChunkGroups`), replacing `AvailableModules`.
- Determines module availability by intersecting its `ChunkGroupInfo`
membership bitmap with the available groups.
- Uses a batch's representative first module, because every module in a
batch has identical group membership.
- Tracks async-loader membership by the parent groups that emit each
loader, preserving nested loader deduplication.
- Scopes availability to its `ModuleGraph`: chunk-group indices only
mean something inside one graph's `ChunkGroupInfo`, so the graph is
stored alongside the bitmap and asserted on every use, and `translate`
re-expresses availability in another graph by mapping indices through
their canonical `ChunkGroupKey`s. Groups the target graph doesn't have
are dropped, which can only duplicate a module, never omit one.
Behavior is unchanged. Availability-dependent asset identifiers are
derived from the *identity* of the available chunk groups — each group
rendered via `ChunkGroupKey::ident_str` (module idents plus merge tags,
merged parents resolved recursively) and sorted — never from the raw
graph-local indices, which are only meaningful within a single
`ModuleGraph` and would collide across the per-endpoint graphs used in
development. Deterministic filenames change as a result; the snapshots
are updated accordingly and the module IDs they contain are identical.
One observable improvement: in Turbopack dev, `/_app` and the page are
chunked in the same graph (#98823), and availability is now tracked by
chunk-group membership, so modules shared by both are loaded only once.
The `// @gate !(turbopack && dev)` that #99910 put on the "loads modules
shared by _app and the page only once" assertion in
`app-document-import-order` is therefore stale, and this PR removes it.
### Verification
- `turbopack-core` — 164 passed
- `next-core` — 84 passed
- Turbopack snapshots — 126 passed
- Nested async availability coverage verifies shared-module and loader
deduplication
- Pages/App development and production suites, including
`404-page-router`, `server-navigation-error`, client-reference chunking,
and full `css-order`
- Turbopack dev `app-document-import-order` (now ungated) and
`pages-app-availability-boundary`, on this PR and on #99050
- Affected Cargo checks and Rust formatting
### Benchmarks
Measured on commits `64b73c08` (this PR) against `c543a06d` (#98823) —
the pre-rebase SHAs of this stack. The stack has since been rebased onto
newer canary; the refactor's code diff is byte-identical across that
rebase, so these numbers still characterise this patch, but they were
not re-measured on the current heads.
The two commits were one apart and differed only in Rust, so the JS
build was shared and the two native binaries were swapped in place with
their sha256 asserted before every run.
`next build --turbopack`, cold (`rm -rf .next`) every run, **3 runs per
side, sides interleaved**. Memory is the peak **aggregate RSS of the
whole build process tree**, sampled from `/proc` every 50 ms (GNU `time
-v` reports only the max of a single waited-for child, which understates
a multi-process build). Median shown, with min–max in brackets.
"disjoint" = the two sides' ranges do not overlap at all.
**Workloads where availability actually engages** — it only does work
when a chunk group inherits availability from a parent, i.e. dynamic
imports or multiple entries:
| Workload | Metric | base | this PR | Δ | |
|---|---|---|---|---|---|
| 200 routes × 6 dynamic imports | build | 108.7 s [105.7–111.6] |
**75.9 s** [74.0–78.0] | **−30.1%** | disjoint |
| | peak RSS | 3462 MB [3361–3467] | **3093 MB** [3052–3153] |
**−10.7%** | disjoint |
| 750 routes, no dynamic imports | build | 314.5 s [313.1–323.0] |
**292.4 s** [289.4–311.2] | **−7.0%** | disjoint |
| | peak RSS | 9220 MB [9212–9687] | 10904 MB [9196–11164] | — |
inconclusive |
**Control workloads** — single entry, no `import()`, so availability is
effectively root everywhere and there is nothing to improve:
| Workload | build Δ | peak RSS Δ | |
|---|---|---|---|
| `bench/module-cost` (~18.6k static modules) | −3.3% | 0.0% | noise |
| `bench/heavy-npm-deps` (1 page, heavy deps) | −1.9% | −1.6% | noise |
Build time improves monotonically with how much availability a workload
exercises (−1.9%, −3.3%, −7.0%, −30.1%). Emitted `.js` file counts are
identical between sides on every workload, consistent with unchanged
behaviour.
Caveats: the dynamic-import workload is synthetic and its 30% figure is
not a typical-app claim; n=3 with no statistical test; aggregate RSS
double-counts pages shared between processes; the 750-route memory
result is bimodal on this PR's side (its own range spans the base
median) and is therefore reported as inconclusive rather than a
regression. The many-pages fixture was reduced from its stock 3000
routes to 750 because 3000 exceeds the 16 GB benchmark machine and
thrashes, which would measure swap rather than chunking.
<!-- NEXT_JS_LLM -->
<!-- fleet 4f67c6f7-bbe2-454b-9a17-822a60ef8f7e -->
Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
