Turbopack: scale default disk write concurrency (#101519)
### What?
On Linux, Turbopack's default disk write concurrency now scales with
available parallelism instead of using a fixed limit of 4. The Linux
default is `clamp(available_parallelism, 4, 16)`, using
`turbo_tasks::parallel::available_parallelism()` so CPU affinity and
`TURBO_TASKS_AVAILABLE_PARALLELISM` are respected.
**macOS, Windows and all other platforms stay at the current fixed
default of 4**, pending measurements on those platforms. Their limits
(e.g. open file limits on macOS, metadata contention on NTFS) differ
from Linux. `TURBO_ENGINE_WRITE_CONCURRENCY` still overrides the default
on every platform.
### Why?
On larger machines, writing many small build outputs was throttled by
the fixed limit. This is part of the build scaling work in #100250.
### How?
- The scaling default is gated with `cfg(target_os = "linux")`. Other
platforms compile to the fixed default of 4.
- On Linux, 4 is the minimum and the fallback when parallelism can't be
determined.
- On Linux, the cap of 16 limits open file handles and blocked tokio
workers. A limit of 32 or 64 did not make builds faster in the sweep
below.
- EMFILE retry handling is unchanged.
Moving writes to `spawn_blocking` is left out. The write path borrows
content and path state and has retry and tracing behavior, so that
change needs its own design and benchmark. In these runs, limits up to
64 did not show a measurable cold-build compile regression, despite IO
currently blocking tokio workers.
### Benchmarks
Setup: same canary base (`4d51724b7c1ce377bce7564e6a5237d2846d0867`),
same 32-thread Linux machine (16 physical cores × 2 SMT), interleaved
batches, native release `next-swc`. `16 CPUs` is `taskset -c 0-15` (8
physical cores + SMT), `32 CPUs` is `0-31`. Results are compile time
(`run-turbopack`) and full `next build` wall time, in seconds, mean ±
sample stddev.
Cold builds delete `.next` before each build. Warm builds restore only a
primed `.next/cache` and delete the rest of `.next`.
`TURBO_ENGINE_IGNORE_DIRTY=1` was set for every run because the locally
built binary came from a dirty tree; without it, the persistent cache
would use a temporary directory.
#### Before/after (compile and wall time)
The build-time before/after runs and the write-concurrency sweep below
were measured with a build whose Linux write path is identical to this
PR's. The peak RSS section uses a build of the exact final code.
`bench/large-app-router` (generated benchmark app from #101502, used
locally only):
| build | CPUs | runs | canary compile | this PR compile | Δ compile |
canary wall | this PR wall |
|---|---:|---:|---:|---:|---:|---:|---:|
| cold | 16 | 4 | 77.42 ± 3.25 | 76.90 ± 5.28 | −0.52 | 109.94 ± 3.91 |
110.08 ± 6.18 |
| cold | 32 | 4 | 55.96 ± 1.88 | 54.48 ± 4.84 | −1.48 | 95.53 ± 3.37 |
92.53 ± 7.25 |
| warm | 16 | 10 | 13.23 ± 0.83 | 12.46 ± 0.80 | **−0.77** | 36.03 ±
1.79 | 35.15 ± 1.82 |
| warm | 32 | 10 | 12.06 ± 0.49 | 11.24 ± 0.81 | **−0.82** | 30.04 ±
1.49 | 29.94 ± 2.50 |
`bench/nested-deps-app-router-many-pages`:
| build | CPUs | runs | canary compile | this PR compile | Δ compile |
canary wall | this PR wall |
|---|---:|---:|---:|---:|---:|---:|---:|
| warm | 16 | 5 | 3.88 ± 0.17 | 2.98 ± 0.05 | **−0.90** | 17.54 ± 0.85 |
16.22 ± 0.24 |
| warm | 32 | 5 | 3.71 ± 0.17 | 2.95 ± 0.18 | **−0.76** | 13.31 ± 0.52 |
12.67 ± 0.55 |
All 10 paired nested-deps warm runs were faster. Cold builds on this
machine are noisy; these cold results show no consistent regression.
#### Peak memory (peak RSS)
Peak RSS is `VmHWM` of the main `next-build` process, which runs
Turbopack and has the largest peak in every run. It is sampled from
`/proc/<pid>/status` every 50 ms; `VmHWM` is a high-water mark, so
sampling only misses growth in the last <50 ms. As a cross-check, the
exact `ru_maxrss` of the reaped process tree
(`getrusage(RUSAGE_CHILDREN)`) agrees within 0–8.5 MiB per run and gives
the same deltas within 1 MiB. Units are GiB, canary vs the final binary
of this PR. Runs are interleaved in random order within each pair. Δ is
the paired mean difference ± its standard error.
`bench/large-app-router`:
| build | CPUs | pairs | canary VmHWM | this PR VmHWM | Δ (paired) |
pairs higher | canary compile (s) | this PR compile (s) |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| cold | 16 | 4 | 13.10 ± 0.13 | 13.22 ± 0.15 | +0.118 ± 0.129 | 3/4 |
78.62 ± 4.81 | 78.94 ± 6.70 |
| cold | 32 | 4 | 13.26 ± 0.17 | 13.22 ± 0.22 | −0.037 ± 0.161 | 1/4 |
58.01 ± 1.63 | 55.86 ± 1.23 |
| warm | 16 | 6 | 5.08 ± 0.01 | 5.09 ± 0.01 | +0.012 ± 0.004 | 5/6 |
13.58 ± 0.70 | 12.10 ± 0.59 |
| warm | 32 | 6 | 4.80 ± 0.01 | 4.80 ± 0.01 | +0.005 ± 0.009 | 3/6 |
12.06 ± 0.67 | 11.14 ± 0.80 |
- **Cold:** the differences are within noise. Run-to-run spread is about
±0.15–0.2 GiB on a ~13.2 GiB peak, with no consistent sign.
- **Warm:** both binaries restore the same canary-primed
persistent-cache snapshot (`TURBO_ENGINE_VERSION` pinned so they share
it). Warm peak RSS depends strongly on the snapshot, because the cache
database is memory-mapped. Two independently primed snapshots differed
by about 0.2 GiB with the same binary. Earlier runs, where each binary
primed its own snapshot, showed a misleading +0.09 GiB that followed the
snapshot, not the binary. That was shown with a 2×2 binary × snapshot
swap.
- **Priming:** snapshots primed with a write limit of 4 vs 16 gave the
same warm peak RSS: 4.79 ± 0.01 vs 4.78 ± 0.01 GiB, across 2 independent
primings each.
- **Concurrency alone:** the canary binary at 16 CPUs (warm, 5 runs
each) gave limit 4: 5.025 ± 0.055 GiB, 8: 4.980 ± 0.003, 16: 4.998 ±
0.022.
- **Remaining difference:** about +12 MiB (+0.24%) at warm/16 CPUs,
consistent in sign (5 of 6 pairs). It is 0 within noise at warm/32 CPUs,
even though 32 CPUs also uses 16 write permits. Cold differences are
within noise but can't resolve a change this small. Whether this counts
as a regression is a reviewer decision.
#### Write-concurrency sweep (canary binary)
Each limit was set with `TURBO_ENGINE_WRITE_CONCURRENCY`.
`bench/large-app-router` cold (3 runs per cell):
| CPUs | 4 | 8 | 16 | 32 | 64 |
|---:|---:|---:|---:|---:|---:|
| 16 compile | 75.20 ± 2.19 | 76.34 ± 3.07 | 76.00 ± 1.41 | 74.84 ± 1.35
| 78.40 ± 4.25 |
| 16 wall | 108.28 ± 3.62 | 110.87 ± 2.86 | 109.17 ± 1.40 | 107.54 ±
1.03 | 111.99 ± 4.14 |
| 32 compile | 63.07 ± 13.48 | 59.43 ± 6.20 | 55.46 ± 4.20 | 55.25 ±
4.42 | 64.83 ± 14.35 |
| 32 wall | 103.30 ± 16.69 | 97.71 ± 8.36 | 93.75 ± 4.68 | 92.92 ± 7.21
| 105.93 ± 20.18 |
`bench/large-app-router` warm (9 runs per cell):
| CPUs | 4 | 8 | 16 | 32 | 64 |
|---:|---:|---:|---:|---:|---:|
| 16 compile | 13.16 ± 0.61 | 12.02 ± 0.56 | 12.03 ± 0.90 | 11.85 ± 0.76
| 12.20 ± 0.59 |
| 16 wall | 35.82 ± 1.90 | 34.92 ± 2.50 | 34.44 ± 1.84 | 35.17 ± 2.26 |
34.84 ± 1.48 |
| 32 compile | 12.37 ± 0.52 | 11.12 ± 0.39 | 11.50 ± 1.16 | 11.47 ± 0.71
| 12.14 ± 1.37 |
| 32 wall | 31.20 ± 1.36 | 29.94 ± 1.83 | 30.47 ± 3.31 | 29.98 ± 2.15 |
30.75 ± 2.59 |
A limit of 4 is clearly the slowest for warm builds. 8 through 32 are
within noise, and 64 trends worse or is noisier. 16 sits in the middle
of that range while keeping the write fan-out bounded. All measurements
are Linux-only, which is why only the Linux default changes; macOS and
Windows remain at 4 until measured there.
### Verification
- `cargo test -p turbo-tasks-fs` (existing tests, 160 passed)
- `cargo clippy -p turbo-tasks-fs --all-targets`
- `cargo fmt -p turbo-tasks-fs -- --check`
- The benchmark runs above, using native release `next-swc` builds
<!-- NEXT_JS_LLM -->
<!-- fleet 2a197d93-f4b0-4541-a604-92e313cb5655 -->
---------
Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
