Compute Turbopack route bundle stats in Rust (#101540)
> [!NOTE]
> Built on #101617 (merged). Alternative approach: #101506.
## What?
Turbopack now computes `.next/diagnostics/route-bundle-stats.json`
itself and emits it as a normal production output asset. This removes
the JS `writeRouteBundleStats` step.
- **Endpoint data:** In production, page endpoints expose the inputs for
their first-load chunks through `EndpointOutput::first_load_chunks`
(`FirstLoadChunks`). These point to data the endpoint already computes:
- **App Router:** client references, client reference chunks and shared
client chunks. Entry chunks use the same CSS and component-chunk
filtering as `entryJSFiles`, followed by the shared root chunks.
- **Pages Router:** the page's client chunk group, plus the `/_app`
chunks.
- **Per-route stats:** `route_bundle_stat` is one turbo-tasks function
per route. It reads the first-load chunks from all of the route's
endpoints, then:
- dedupes chunks by path in endpoint order;
- sums their `Asset::size()` values.
- **Endpoint lists:**
- Pages pages list their HTML endpoint followed by `/_app`.
- App routes list all of their HTML endpoints.
- **Excluded routes:** Pages Router API routes and App Router route
handlers ship no client JS, so they have no row.
- **Chunk size:** the new default `Asset::size()` returns
`Vc<Option<u64>>`, reusing `AssetContent::len()`. Missing files and
redirects have no size, so they aren't counted.
- **Emission:** the asset is included in `output_assets_operation` for
the full build or deferred phase, and written by
`emit_all_output_assets` like other diagnostic files. Nothing runs or
writes files after emission.
- **Paths:** forward slashes on all platforms, matching #101617.
- **Output difference from canary:** canary's JS implementation reported
Pages API routes with the `/_app` chunks. This PR omits them. The other
routes and the sort order are unchanged.
- **Unchanged:** webpack and `next dev` still write no stats file.
## Why?
This continues the effort to make Turbopack `next build` scale (#100250,
item 60). Canary's JS step `require()`s every page's client reference
manifest on the main thread just to read the entry JS files. Turbopack
already holds that data in memory.
## Performance
Benchmark: synthetic `bench/large-app-router` with 5,102 routes; the
fixture is not part of this PR. These measurements were taken at
`197ab62c`, before the rename, test cleanup and Pages API route
exclusion. I did not rerun the benchmark for those changes; they don't
affect this fixture's build work.
- Both sides use canary `bf3d38e4` and release next-swc, on the same
machine.
- Builds were cold and interleaved, with 3 runs per side at each CPU
count. Values are mean ± sample stddev.
- Peak RSS was measured with GNU `time`.
- `TURBO_ENGINE_IGNORE_DIRTY=1` was set for both sides.
| CPUs | Side | `next build` wall (s) | Turbopack compile (s) | JS stats
step (s) | Peak RSS (GiB) |
|---:|---|---:|---:|---:|---:|
| 16 | canary | 97.26 ± 1.88 | 69.25 ± 1.44 | 2.87 ± 0.08 | 14.10 ± 0.18
|
| 16 | this PR | 94.28 ± 8.08 | 69.02 ± 7.16 | – | 13.96 ± 0.06 |
| 32 | canary | 76.83 ± 5.06 | 49.79 ± 3.31 | 2.79 ± 0.15 | 14.46 ± 0.29
|
| 32 | this PR | 73.86 ± 0.72 | 49.74 ± 0.71 | – | 14.53 ± 0.21 |
- **Output:** at `197ab62c`, `route-bundle-stats.json` was
byte-identical in all 12 runs.
- **Wall time:** 2.98s faster at both 16 and 32 CPUs.
- **Compile time:** unchanged (−0.23s at 16 CPUs, −0.05s at 32 CPUs).
- **Peak memory:** within variation (−0.14 GiB at 16 CPUs, +0.07 GiB at
32 CPUs).
- **Stats cost:** the Rust stats run inside emission, so they have no
separate span.
An earlier version of this PR computed the stats in a separate operation
after emission. That regressed wall time by 7–10s because turbo-tasks
had to reactivate and re-aggregate finished endpoint subgraphs.
## Tests
`test/production/route-bundle-stats` covers:
- component chunks off and on;
- App routes, compared against their `entryJSFiles` + `rootMainFiles`;
- a parallel-slot route, checking that both pages exist and that the
stats include the ordered union of their entry JS chunks;
- Pages routes, compared against `build-manifest.json` pages + `/_app`;
- an excluded Pages API route and an excluded App route handler;
- a custom `distDir`;
- exact bytes, ordering and forward-slash paths.
Each variant gets its own `nextTestSetup` and `nextConfig`, instead of
patching `next.config.js`. App Router manifests are read with
`getClientReferenceManifest` from `next-test-utils`, and the test
asserts that `entryJSFiles` is present. Deployments are skipped with
`@force-gate !deploy`, since they have no local build output.
The parallel-slot pages share one loader tree, so their entry JS chunks
are currently identical. The test protects the union/deduplication
behavior but does not demonstrate a byte difference between the old and
new endpoint selection.
## Verification
- `test/production/route-bundle-stats` passes 15/15 with Cache
Components off and on, using release next-swc built on this branch.
- Before the Pages API route change, the updated test fails in all three
variants because `/api/hello` is still reported.
- `cargo clippy` and `cargo fmt --check` pass for `next-core`,
`next-api` and `next-napi-bindings`. Prettier, eslint and `pnpm
typescript` pass.
- On the mixed App/Pages fixture:
- normal and `--experimental-app-only` builds emit forward-slash stats;
- webpack writes no stats file;
- `next dev` serves pages without one.
- At `197ab62c`, before the Pages API route exclusion, the mixed fixture
was byte-identical to the previous revision in both modes, and the
parallel-slot row was unchanged.
Not covered yet: dedicated parity tests for strict route matching,
intercepted routes, Edge pages, deferred entries and a `distDir` outside
the project.
<!-- NEXT_JS_LLM -->
<!-- fleet 8793ade5-4b7e-45ac-9d71-dd78d2669301 -->
---------
Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>