turbopack: use Bitset for persisted chunk group sets (#100630)
Stacked on #100568, the top of the `Bitset` stack: #100566 (crate) →
#100643 → #100567 → #100568 → this.
## Summary
The PRs below move the module-graph traversals onto `Bitset`, but they
still convert to `RoaringBitmapWrapper` at the boundary, and persisted
values and production chunking stay Roaring. This PR finishes that,
using the bincode encoding and set operations that #100566 adds:
- **turbo-tasks** depends on `turbo-bitset` and implements `TaskInput`,
`NonLocalValue` and `ValueDebugFormat` for `Bitset`, next to its other
foreign-type impls.
- **turbopack-core** drops `RoaringBitmapWrapper` and the `roaring`
dependency:
- `ModuleToChunkGroups`, `ChunkingHeuristicsInfo::priority_routes`, the
module and chunk-item batch types, and production chunking use `Bitset`
directly.
- So do the local sets: `inherits_from`, cluster propagation,
`cluster_groups` and `pairs_with`.
- `Bitset::to_roaring()` (and its tests and benchmark) is removed, and
`roaring` becomes a dev-dependency of `turbo-bitset` (for the benchmarks
and the `sizes` example).
- `next-api` drops its unused `roaring` dependency, which the
workspace's unused-dependency lint reports once `turbopack-core` no
longer uses it.
**Cache compatibility:** the persisted encoding and value hashes of
these types change (task inputs are hashed with fixed-width integers,
for which the `Bitset` encoding always uses the bitmap form). The
persistent cache is versioned by `git describe`
(`turbo-tasks-backend/src/database/db_versioning.rs`), so normal builds
won't read old entries. With `TURBO_ENGINE_VERSION`,
`TURBO_ENGINE_DISABLE_VERSIONING` or `TURBO_ENGINE_IGNORE_DIRTY` in a
dirty checkout, clear the cache when switching between binaries with and
without this change.
## Results
> **Historical, not rerun:** measured in the previous review round,
before this stack was rebased onto canary `502b6b7d` and before the
latest review changes on `b149cf8a` (#100566), `c7ea9063` (#100643),
`e322a1c1` (#100567), `6536eeea` (#100568) and `417a41fd` (#100630).
This round only reran the `turbo-bitset` microbenchmarks in #100566.
Base is #100568's head, so the encoding of #100566 is the same on both
sides; only what this PR persists as `Bitset` instead of Roaring
differs.
### Full builds
Full `next build` (including prerendering) with uninstrumented
`next-swc` built with the repository's `release` profile (thin LTO), on
a shared VM with 8 vCPUs and 16 GB. 5 randomized, interleaved runs per
variant and app, median with min–max. Each run:
1. **cold**: removes `.next` and builds;
2. **cache**: measures `.next/cache` (the Turbopack build cache) with
`du -sb`;
3. **warm**: removes everything in `.next` except that variant's own
cache and builds again.
Caches are never shared between variants. Both binaries were built from
a checkout with uncommitted changes, so every build ran with
`TURBO_ENGINE_IGNORE_DIRTY=1` to keep the filesystem cache enabled (each
build logs that it is).
| full `next build` | base (#100568) | this PR |
|---|---|---|
| `bench/nested-deps-app-router-many-pages` (3,000 routes): cold build |
53.1 s (51.7–54.1) | 52.2 s (51–56.5) |
| `bench/nested-deps-app-router-many-pages` (3,000 routes):
`.next/cache` size | 762.95 MB (762.85–763.19) | 761.49 MB
(761.24–761.84) |
| `bench/nested-deps-app-router-many-pages` (3,000 routes): warm build
(own cache) | 35.8 s (34.8–36.7) | 35.5 s (34.8–36.6) |
| generated 300-route app: cold build | 11.3 s (10.8–11.7) | 11.1 s
(10.9–11.7) |
| generated 300-route app: `.next/cache` size | 213.84 MB
(213.45–214.45) | 213.85 MB (213.76–213.94) |
| generated 300-route app: warm build (own cache) | 5.42 s (5.14–5.5) |
5.46 s (5.18–5.57) |
All 40 builds succeeded with the same routes per app. The cache is
reused: warm builds take 48–68% of the cold time for both variants and
rewrite part of the cache. On `many-pages` the cache is 1.46 MB (0.19%)
smaller with non-overlapping ranges; on the generated app the difference
is within the run-to-run variation (±0.5 MB). Cold and warm times
overlap, so this makes no end-to-end speed claim.
<details>
<summary>Charts: full builds (linear)</summary>
Median wall time in seconds. Ranges overlap; exact values are in the
table above.
```mermaid
xychart-beta
title "bench/nested-deps-app-router-many-pages: full build, median s"
x-axis ["cold", "warm"]
y-axis "s"
bar "base (#100568)" [53.1, 35.8]
bar "this PR" [52.2, 35.5]
```
```mermaid
xychart-beta
title "generated 300-route app: full build, median s"
x-axis ["cold", "warm"]
y-axis "s"
bar "base (#100568)" [11.3, 5.42]
bar "this PR" [11.1, 5.46]
```
</details>
### Encoded sizes
Logical set payloads from local-only size probes (bytes of the encoded
values, not the cache database, so keys, metadata and compression aren't
included), same compile-mode builds as below. They are identical in
every run:
| logical payload | base (#100568) | this PR |
|---|---|---|
| `bench/nested-deps-app-router-many-pages` (3,000 routes): encoded
`ModuleToChunkGroups` values (33,779 sets) | 40,212,209 B | **9,711,974
B** (-76%) |
| `bench/nested-deps-app-router-many-pages` (3,000 routes): encoded
`ModuleBatch` chunk groups (4,132 sets) | 721,522 B | **97,208 B**
(-87%) |
| `bench/nested-deps-app-router-many-pages` (3,000 routes): encoded
`ModuleBatchGroup` chunk groups (3,021 sets) | 138,211 B | **27,960 B**
(-80%) |
| `bench/nested-deps-app-router-many-pages` (3,000 routes): per-module
set p50 / p99 / max | 19 / 6,023 / 8,211 B | **5 / 1,006 / 2,501 B** |
| generated 300-route app: encoded `ModuleToChunkGroups` values (12,240
sets) | 1,971,644 B | **434,660 B** (-78%) |
| generated 300-route app: encoded `ModuleBatch` chunk groups (2,101
sets) | 220,751 B | **42,221 B** (-81%) |
| generated 300-route app: encoded `ModuleBatchGroup` chunk groups (629
sets) | 30,165 B | **5,578 B** (-82%) |
| generated 300-route app: per-module set p50 / p99 / max | 19 / 1,241 /
3,687 B | **3 / 235 / 339 B** |
### Stage times
`next build --experimental-build-mode compile` with release `next-swc`
(LTO off, 16 codegen units) built from each PR's head plus local-only
stage timers. Median of 5 randomized, interleaved runs per variant,
min–max in parentheses, in ms. All five layers of the stack were
measured in the same interleaved set.
| stage (ms) | previous PR | this PR |
|---|---|---|
| `bench/nested-deps-app-router-many-pages` (3,000 routes): heuristics +
building the result | 37.7 (37.1–39.4) | **5.17 (4.66–5.39)** |
| `bench/nested-deps-app-router-many-pages` (3,000 routes): production
chunk merging (summed) | 317 (313–326) | **249 (240–253)** |
| generated 300-route app: heuristics + building the result | 3.71
(3.57–3.78) | **0.62 (0.574–0.657)** |
| generated 300-route app: production chunk merging (summed) | 19.3
(18.7–20.5) | **14.2 (13.6–14.8)** |
<details>
<summary>Charts: stage times (linear)</summary>
Median ms per stage; the first series is the previous PR. Exact values
and ranges are in the table above.
```mermaid
xychart-beta
title "bench/nested-deps-app-router-many-pages, median ms"
x-axis ["heuristics + building the result", "production chunk merging (summed)"]
y-axis "ms"
bar "previous PR" [37.7, 317]
bar "this PR" [5.17, 249]
```
```mermaid
xychart-beta
title "generated 300-route app, median ms"
x-axis ["heuristics + building the result", "production chunk merging (summed)"]
y-axis "ms"
bar "previous PR" [3.71, 19.3]
bar "this PR" [0.62, 14.2]
```
</details>
Sum of the measured module-graph stages per run (chunk group info and
building its result, collect, merged modules and interning), whole
stack:
| sum of module-graph stages (ms) | #100566 (before the usage PRs) | top
of stack (#100630) |
|---|---|---|
| `bench/nested-deps-app-router-many-pages` (3,000 routes) | 750
(746–767) | **102 (99.7–113)** |
| generated 300-route app | 36.3 (35.1–38) | **17.8 (17.5–18.1)** |
<details>
<summary>Charts: stage sum per layer (linear)</summary>
Median ms of the summed module-graph stages, one bar per layer of the
stack. Exact values for the ends are in the table above.
```mermaid
xychart-beta
title "bench/nested-deps-app-router-many-pages: stage sum per layer, median ms"
x-axis ["#100566", "#100643", "#100567", "#100568", "#100630"]
y-axis "ms"
bar "stage sum" [750, 455, 288, 128, 102]
```
```mermaid
xychart-beta
title "generated 300-route app: stage sum per layer, median ms"
x-axis ["#100566", "#100643", "#100567", "#100568", "#100630"]
y-axis "ms"
bar "stage sum" [36.3, 28.8, 27.1, 21, 17.8]
```
</details>
Visit counts and the emitted output are identical across all layers on
both apps (14,028 and 2,142 files, same digests).
## Verification
- `cargo test -p turbo-bitset`, `-p turbo-tasks --lib`, `-p
turbopack-core --lib`, the turbopack snapshot tests (no snapshot
changes) and execution tests.
- `cargo fmt --check`, `cargo clippy` (touched crates plus `next-api`,
`next-core`, all targets) and `cargo doc` with `-D warnings` (no
unused-dependency warning).
- `cargo tree -p turbopack-core -e normal` no longer contains `roaring`.
- The builds above, with timers and probes kept local only.
<!-- NEXT_JS_LLM -->
<!-- fleet b56269ed-2145-4f30-b0fe-dc9762520521 -->
---------
Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
Co-authored-by: Vercel Fleet <fleet@vercel.com>
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
