turbo
445b0c70 - perf: Overlap external dependency hashing with package file hashing (#13234)

Commit
114 days ago
perf: Overlap external dependency hashing with package file hashing (#13234) ## Why Between the package graph finishing and the first task dispatching, two chunks of hashing work run serially that don't need to: 1. `TaskHasher` construction computes external dependency hashes for every package (~47ms on a 1191-package monorepo) *after* the parallel `hash_scope` block has already finished, even though it depends only on the package graph. 2. `calculate_file_hashes` deduplicates file hashing by unique `(package, inputs)` key, but then recomputes the summary hash *per task* when distributing results — 1866 hash computations where ~1200 suffice (~20ms of the previously uninstrumented `hash_scope` tail). Both sit directly on the time-to-first-task critical path. ## What Three commits, separable if a reviewer wants: 1. `chore:` phase spans (`collect_task_hash_keys`, `hash_unique_inputs`, `distribute_task_file_hashes`) inside `calculate_file_hashes` — this instrumentation is how the per-task re-hashing was found, and it keeps the phase visible in profiles. 2. `perf:` compute each unique key's summary hash once, inside the existing parallel phase; distribution becomes pure map inserts. 3. `perf:` move external dependency hashing into the existing 4-way `rayon::scope` in `Run::run_with_opts`, passing the precomputed cache into `Visitor::new` (which falls back to the old serial path if not provided). The per-task fallback for cache misses is unchanged. ## How to verify On a 1191-package internal-scale monorepo (4-core machine, interleaved pairs vs main, telemetry disabled): - Time-to-first-task (real run): 1515.8ms → 1481.3ms median, 4 of 5 pairs improved. - Span timeline: the serial 46.6ms `visitor_new` block disappears; `hash_scope` absorbs the work (78ms → 97.5ms wall on 4 cores, so net ~−27ms there — the win grows with core count since the added work is fully parallel). `distribute_task_file_hashes` drops 20.1ms → ~2ms. - `--dry=json` output byte-identical to main (modulo run id) on two large monorepos — task hashes are unchanged by construction. - 431 turborepo-lib + 14 turborepo-task-hash tests pass; clippy and fmt clean.
Author
Parents
Loading