turborepo
b3c0f46d - perf: Parallelize task hash computation across topological waves (#11969)

Commit
22 hours ago
perf: Parallelize task hash computation across topological waves (#11969) ## Summary Moves task hash computation out of the serial dispatch loop and into a parallel pre-computation phase using rayon. Tasks are grouped into topological waves by dependency depth, and each wave is hashed concurrently. Also parallelizes external dependency hash precomputation. Previously, every task's hash and execution environment were computed one-at-a-time inside the `while let` visitor dispatch loop. With ~1700 tasks at ~160μs each, this was ~270ms of serial work on the critical path. Now, the same work completes in ~53ms using rayon's thread pool. Also cleans up `Engine::dependencies()`/`dependents()` to return `Vec` instead of `HashSet`, and `calculate_dependency_hashes` to take a `&[&TaskNode]` slice, since all callers only iterate — no set operations are performed. ## Benchmarks Profiled with `--profile` across three monorepos of varying size (full cache hit scenario, median of 5 warm runs): | Repo | Tasks | Baseline | After | Delta | |------|-------|----------|-------|-------| | Small (~5 packages) | 15 | 170ms | 169ms | ~flat | | Medium (~125 packages) | 700 | 876ms | 876ms | ~flat | | Large (~1000 packages) | 1700 | 867ms | 775ms | **-92ms (-10.6%)** | `queue_task` self-time in the large repo dropped from **268ms to 78ms** (71% reduction). The improvement scales with task count. ## Changes - **`crates/turborepo-lib/src/task_graph/visitor/mod.rs`** — Added `precompute_task_hashes()` which groups tasks into topological waves and hashes each wave in parallel via rayon. The dispatch loop now looks up pre-computed hashes instead of computing them inline. - **`crates/turborepo-engine/src/lib.rs`** — `dependencies()`, `dependents()`, and `neighbors()` return `Vec` instead of `HashSet`. Updated `EngineInfo` impl's associated type accordingly. - **`crates/turborepo-task-hash/src/lib.rs`** — `calculate_task_hash` takes `&[&TaskNode]` instead of `HashSet<&TaskNode>`. `calculate_dependency_hashes` takes a slice. `precompute_external_deps_hashes` uses `par_iter` instead of a sequential loop.
Author
Parents
Loading