next.js
d489c0e7 - Compute Turbopack route bundle stats in Rust (#101540)

Commit
6 hours ago
Compute Turbopack route bundle stats in Rust (#101540) > [!NOTE] > Built on #101617 (merged). Alternative approach: #101506. ## What? Turbopack now computes `.next/diagnostics/route-bundle-stats.json` itself and emits it as a normal production output asset. This removes the JS `writeRouteBundleStats` step. - **Endpoint data:** In production, page endpoints expose the inputs for their first-load chunks through `EndpointOutput::first_load_chunks` (`FirstLoadChunks`). These point to data the endpoint already computes: - **App Router:** client references, client reference chunks and shared client chunks. Entry chunks use the same CSS and component-chunk filtering as `entryJSFiles`, followed by the shared root chunks. - **Pages Router:** the page's client chunk group, plus the `/_app` chunks. - **Per-route stats:** `route_bundle_stat` is one turbo-tasks function per route. It reads the first-load chunks from all of the route's endpoints, then: - dedupes chunks by path in endpoint order; - sums their `Asset::size()` values. - **Endpoint lists:** - Pages pages list their HTML endpoint followed by `/_app`. - App routes list all of their HTML endpoints. - **Excluded routes:** Pages Router API routes and App Router route handlers ship no client JS, so they have no row. - **Chunk size:** the new default `Asset::size()` returns `Vc<Option<u64>>`, reusing `AssetContent::len()`. Missing files and redirects have no size, so they aren't counted. - **Emission:** the asset is included in `output_assets_operation` for the full build or deferred phase, and written by `emit_all_output_assets` like other diagnostic files. Nothing runs or writes files after emission. - **Paths:** forward slashes on all platforms, matching #101617. - **Output difference from canary:** canary's JS implementation reported Pages API routes with the `/_app` chunks. This PR omits them. The other routes and the sort order are unchanged. - **Unchanged:** webpack and `next dev` still write no stats file. ## Why? This continues the effort to make Turbopack `next build` scale (#100250, item 60). Canary's JS step `require()`s every page's client reference manifest on the main thread just to read the entry JS files. Turbopack already holds that data in memory. ## Performance Benchmark: synthetic `bench/large-app-router` with 5,102 routes; the fixture is not part of this PR. These measurements were taken at `197ab62c`, before the rename, test cleanup and Pages API route exclusion. I did not rerun the benchmark for those changes; they don't affect this fixture's build work. - Both sides use canary `bf3d38e4` and release next-swc, on the same machine. - Builds were cold and interleaved, with 3 runs per side at each CPU count. Values are mean ± sample stddev. - Peak RSS was measured with GNU `time`. - `TURBO_ENGINE_IGNORE_DIRTY=1` was set for both sides. | CPUs | Side | `next build` wall (s) | Turbopack compile (s) | JS stats step (s) | Peak RSS (GiB) | |---:|---|---:|---:|---:|---:| | 16 | canary | 97.26 ± 1.88 | 69.25 ± 1.44 | 2.87 ± 0.08 | 14.10 ± 0.18 | | 16 | this PR | 94.28 ± 8.08 | 69.02 ± 7.16 | – | 13.96 ± 0.06 | | 32 | canary | 76.83 ± 5.06 | 49.79 ± 3.31 | 2.79 ± 0.15 | 14.46 ± 0.29 | | 32 | this PR | 73.86 ± 0.72 | 49.74 ± 0.71 | – | 14.53 ± 0.21 | - **Output:** at `197ab62c`, `route-bundle-stats.json` was byte-identical in all 12 runs. - **Wall time:** 2.98s faster at both 16 and 32 CPUs. - **Compile time:** unchanged (−0.23s at 16 CPUs, −0.05s at 32 CPUs). - **Peak memory:** within variation (−0.14 GiB at 16 CPUs, +0.07 GiB at 32 CPUs). - **Stats cost:** the Rust stats run inside emission, so they have no separate span. An earlier version of this PR computed the stats in a separate operation after emission. That regressed wall time by 7–10s because turbo-tasks had to reactivate and re-aggregate finished endpoint subgraphs. ## Tests `test/production/route-bundle-stats` covers: - component chunks off and on; - App routes, compared against their `entryJSFiles` + `rootMainFiles`; - a parallel-slot route, checking that both pages exist and that the stats include the ordered union of their entry JS chunks; - Pages routes, compared against `build-manifest.json` pages + `/_app`; - an excluded Pages API route and an excluded App route handler; - a custom `distDir`; - exact bytes, ordering and forward-slash paths. Each variant gets its own `nextTestSetup` and `nextConfig`, instead of patching `next.config.js`. App Router manifests are read with `getClientReferenceManifest` from `next-test-utils`, and the test asserts that `entryJSFiles` is present. Deployments are skipped with `@force-gate !deploy`, since they have no local build output. The parallel-slot pages share one loader tree, so their entry JS chunks are currently identical. The test protects the union/deduplication behavior but does not demonstrate a byte difference between the old and new endpoint selection. ## Verification - `test/production/route-bundle-stats` passes 15/15 with Cache Components off and on, using release next-swc built on this branch. - Before the Pages API route change, the updated test fails in all three variants because `/api/hello` is still reported. - `cargo clippy` and `cargo fmt --check` pass for `next-core`, `next-api` and `next-napi-bindings`. Prettier, eslint and `pnpm typescript` pass. - On the mixed App/Pages fixture: - normal and `--experimental-app-only` builds emit forward-slash stats; - webpack writes no stats file; - `next dev` serves pages without one. - At `197ab62c`, before the Pages API route exclusion, the mixed fixture was byte-identical to the previous revision in both modes, and the parallel-slot row was unchanged. Not covered yet: dedicated parity tests for strict route matching, intercepted routes, Edge pages, deferred entries and a `distDir` outside the project. <!-- NEXT_JS_LLM --> <!-- fleet 8793ade5-4b7e-45ac-9d71-dd78d2669301 --> --------- Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Author
Parents
Loading