next.js

0b9fffb1 - Turbopack: add turbo-trace-size CLI to inspect trace file size (#99765)

Turbopack: add turbo-trace-size CLI to inspect trace file size (#99765) ### What? Adds `turbopack/crates/turbopack-trace-size`, a CLI (`turbo-trace-size`) that shows where the bytes in a Turbopack trace file (`.next-profiles/trace-turbopack.bin`) go. It decodes the postcard `TraceRow` stream (raw, gzip or zstd) and reports: - size per row type (`Start`, `Enter`, `Exit`, `AllocationCounters`, ...) - the components of `Start`/`Event`/`Record` rows (name, target, values, fixed fields) - bytes per span name, including its Enter/Exit/End/Record rows and Events - bytes per attribute key - the bytes spent re-writing the same span names, targets, keys and string values ```sh cargo run --bin turbo-trace-size --release -- .next-profiles/trace-turbopack.bin --top 30 ``` ### Why? `NEXT_TURBOPACK_TRACING` traces get large quickly. This tool shows which events cause that, so we can decide what to slim down. ### Findings: `bench/heavy-npm-deps`, `NEXT_TURBOPACK_TRACING=1 next build` 68.42 MiB raw, 4.66 M rows (gzip -1: 21.3 MiB, zstd -3: 21.2 MiB): | row type | count | bytes | avg | % | | ------------------ | --------: | --------: | -----: | ----: | | AllocationCounters | 2,280,168 | 45.00 MiB | 20.7 B | 65.8% | | Exit | 1,140,084 | 8.40 MiB | 7.7 B | 12.3% | | Enter | 1,140,084 | 8.40 MiB | 7.7 B | 12.3% | | Start | 39,481 | 5.57 MiB | 148 B | 8.1% | | Record | 21,913 | 804 KiB | 38 B | 1.1% | | End | 39,480 | 265 KiB | 7 B | 0.4% | | MemorySample | 244 | 2.5 KiB | 10 B | 0.0% | | Event | 8 | 368 B | 46 B | 0.0% | - **AllocationCounters take 66% of the file.** `RawTraceLayer` writes one before every `Enter` and every `Exit`, so there are exactly 2× as many as Enter rows. Each one holds an absolute timestamp, the thread id and 4 cumulative counters. - **Enter/Exit take another 25%.** Each span is entered about 29 times on average, because async spans are re-entered on every poll. `module` (module_graph) and `process module` are the biggest, at about 6.3–6.5 MiB each. - **Start rows are 8%**, mostly attribute values: `name` (2.6 MiB), `lookup_path` (1 MiB), `reference_type` (0.6 MiB). - **Repeated span names, targets and keys** come from only 147 distinct strings. At most about 3% could be saved by writing each one once, before the cost of referencing them. String values could save at most another 4%. So skipping or delta-encoding the allocation counters, and delta-encoding Enter/Exit timestamps, would cut the size far more than interning strings would. That is left for a follow-up. Note: the trace was captured with the published `@next/swc-linux-x64-gnu@16.4.0-canary.31` native binary that `pnpm install` put in place, running with JS built from this checkout (16.5.0-canary.1). It was not a source-built native binding. ### How? - The crate reuses `TraceRow`/`TraceValue` from `turbopack-trace-utils` and reads the stream incrementally. Every row byte is counted, so rows + header add up exactly to the decompressed size. - Rows are written from per-thread buffers, so Enter/Exit/End rows can show up before the `Start` of their span. Like the trace server, the tool queues those rows until the `Start` arrives and keeps id mappings after `End`. - `contributing/turbopack/tracing.md` now documents the tool. ### Verification - `cargo test -p turbopack-trace-size`: unit tests check byte accounting against postcard, gzip (multi-member) and zstd input, rows split across reads, headerless files, an incomplete last row, invalid rows, cross-thread ordering and reused ids. - `cargo clippy -p turbopack-trace-size --all-targets -- -D warnings` - Ran it on the trace above: raw, gzip and zstd copies give identical breakdowns, and rows + header = 71,739,555 B, the exact file size. <!-- NEXT_JS_LLM --> <!-- fleet 4a52842b-f1a2-43cb-9779-3655caada6a0 --> Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>