next.js
3cdbd0f2 - Turbopack: delta-encode trace timestamps (#99809)

Commit
29 days ago
Turbopack: delta-encode trace timestamps (#99809) ## Summary Delta-encodes all timestamps in Turbopack trace files. Absolute microsecond timestamps take 4–5 bytes as varints. The difference to the previous row of the same thread usually fits in 1–2 bytes. - **Format:** every row timestamp (`Start`, `End`, `Enter`, `Exit`, `Event`, `MemorySample`) is now the difference to the previous timestamp in the same thread local buffer. The serialized `ts` fields are `DeltaEncodedTimestamp`, an `i64` newtype that postcard zigzag-encodes, so a clock that goes backwards still works. `TraceRow` is generic over its timestamp type: `TraceRow<'a>` is the serialized form, and decoding returns `TraceRow<'a, u64>` with absolute timestamps, so the types show whether a row still needs decoding. - **New row:** `TimestampBase { ts }` carries an absolute timestamp. The writer puts one before the first row with a timestamp in every buffer. Buffers are flushed independently and interleave in the file, so each one can be decoded on its own. - **Large jumps:** a new base is also written when a difference doesn't fit in an `i64`. - **Writer:** the encoder state lives in the buffer and is reset when a buffer is sent, stolen by the writer thread or reused. Removing the last row (#99780) also removes its base and restores the encoder state. - **Readers:** the trace server, `turbo-trace-size` and the tests share `TimestampDecoder`, which keeps its state across incremental reads. A timestamp before any base is an error. `turbo-trace-size` reports the error with its offset. The header is bumped to `TRACEv1` in #99781, at the top of this format-changing stack. Stack (on top of #99765): 1. #99777: `no-memory` flag 2. #99778: Merge allocation counters into Enter/Exit rows 3. #99779: Delta-encode allocation counters, skip unchanged ones 4. #99780: Omit Exit+Enter pairs with (almost) no gap 5. #99808: zstd flags 6. #99809: **This PR**: delta-encoded timestamps 7. #99781: Bump the trace header to `TRACEv1` 8. #99810: `upload-trace` accepts compressed traces ## Measurements `next build` of `bench/heavy-npm-deps`, release bindings built from the previous PR and from this one. Medians of 3 rounds that alternate between the two builds: | mode | before | after | change | | --- | --- | --- | --- | | `1` | 23.94 MB | 21.03 MB | −12.2% | | `1,zstd` | 6.15 MB | 4.72 MB | **−23.2%** | | `turbo-tasks` | 155.97 MB | 145.28 MB | −6.9% | | `turbo-tasks,zstd` | 28.14 MB | 22.13 MB | **−21.4%** | - **Row sizes:** Enter, Exit and End rows get about 2.2 bytes smaller each. - **Overhead:** `TimestampBase` rows add 137 B (35 rows) to the `1` trace and 896 B (214 rows) to the `turbo-tasks` trace. ## Verification - **Tests:** - encoder and decoder round trips, including large jumps and incremental reads; - buffers that are sent, stolen or reused, and a buffer that starts with a `Record` (its base comes before the first row with a timestamp); - removing the first or last row of a buffer; - exact timestamps across 2 threads and many buffers. - **Captures:** the release-build captures are decoded fully by `turbo-trace-size` and load completely in the trace server, raw and zstd. Span counts per name match the previous PR. <!-- NEXT_JS_LLM --> <!-- fleet 4a52842b-f1a2-43cb-9779-3655caada6a0 --> --------- Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Author
Parents
Loading