next.js

44525146 - turbo-tasks: lock-free priority queue for lowest-priority tasks, skip heap lock when empty (#99773)

turbo-tasks: lock-free priority queue for lowest-priority tasks, skip heap lock when empty (#99773) ### What? Removes the global mutex from the turbo-tasks `PriorityRunner` ready queue for the hot paths, and makes `Queue::pop()` skip the remaining heap mutex entirely when the heap is empty. Supersedes #94841: this is that PR rebased onto canary (keeping the keyed claim / inline execution from #96808), plus the empty-heap fast path, plus two simplification passes and review fixes. ### Why? In a from-scratch build, nearly every scheduled task has the lowest priority (`TaskPriority::Initial`). Before this change, every `schedule` and every worker `pop` went through one `Mutex<Queue>`, which becomes a contention point at high worker counts. ### How? **Queue split by band** - `Executor<C, T, P>` declares `const LOWEST_PRIORITY: P` (`TaskPriority::Initial` for turbo-tasks). Items with that priority go to a lock-free MPMC `ConcurrentQueue` (FIFO; all items share a priority). - All other priorities go to a `Mutex<BinaryHeap>` ordered by `P: Ord`. `pop` drains the heap first, then the lowest band. - The heap starts with capacity 128 and only shrinks back to 128 when it becomes empty. - Lock ordering: `pop` may take a `claimable` shard lock while holding the heap mutex; nothing takes the heap mutex while holding a shard lock. **Scheduler / worker handshake** - `schedule` takes no global lock. Liveness (no task stranded when the last worker retires) is a store-buffering pattern: the scheduler pushes then reads `active_workers`; a retiring worker decrements `active_workers` then re-checks the queue. Both sides put a `fence(SeqCst)` between their write and their read, so one of them always observes the other. - The saturated hot path is push → `fence(SeqCst)` → `load`, so schedulers no longer write the shared `active_workers` cache line. Every worker that gives up its count goes through `decrease_active_workers` (`fetch_sub(AcqRel)` → `fence(SeqCst)` → re-check). **One queued task per key** - `Claimable::claim_key()` returns `Self::Key` (no `Option`): every task has a key. - Queued tasks are stored by value in `claimable: FxDashMap<T::Key, (P, T)>`. The heap and the `ConcurrentQueue` only store `T::Key`. - Pushing a key that is still queued **replaces** the queued task. The replaced task goes to the new `Executor::discard` hook (default: no-op). `TurboTasksExecutor::discard` balances the foreground job that `TurboTasks::schedule` began for it. - This only applies to queued tasks: a task that `schedule` spawns directly never enters the queue. - Claim = `claimable.remove(key)`. Pop = `claimable.remove_if(key, priority matches the band entry)`. Stale band entries are tombstones that a pop skips; a stale entry never takes a task queued again with a different priority. With the *same* priority it may take the newer task early, which only reorders equal-priority tasks. - No per-task allocation or per-task mutex, no new `unsafe`. Keyed pushes/pops take a brief DashMap shard lock. **Unique local-task keys** - `ExecutionId` is a wrapping `u16`, so `ScheduleKey::LocalTask(ExecutionId, LocalTaskId)` was not unique across concurrent executions; with replacement that hung the build. - The key is now `ScheduleKey::LocalTask(Option<TaskId>, ExecutionId, LocalTaskId)`. The remaining exception is two concurrent top-level executions (no parent task) sharing a wrapped `ExecutionId`. - Local tasks stay in the runner (not plain tokio tasks) so readers can keep claiming them for inline execution. **Empty-heap fast path** - `heap_non_empty: AtomicBool` is written only under the heap lock. `pop()` checks it and goes straight to the lowest band when the heap is empty, so the common cold-build pop never touches the heap mutex. - A stale `true` costs only an unnecessary lock. A stale `false` can't hide a push that must be observed, by the same fence handshake. ### Tests - `priority_runner` tests use plain `u32` priorities. New: `heap_only_shrinks_when_empty`. The saturated-path regression test is now `stress_saturated_schedule_vs_retiring_worker`. - Replacement within and across bands, claims in both bands, stale entries, concurrent same-key producers/poppers/claimer, claim-vs-pop exactly-once, band ordering, heap-flag drain/refill, pop not waiting on the empty heap's lock, three liveness stress tests. - Stress tests use a timer-only tokio runtime: with `enable_all()` on `wasm32-wasip1-threads`, newly spawned tasks reproducibly stopped being polled. - On canary `b24905530`: `turbo-tasks` 134 passed (two full runs), `turbo-tasks-backend` 282, wasi lib 110 passed, `priority_runner` 5× repeated, clippy `-D warnings`, fmt. Cold 32-core `next build` of `bench/nested-deps-app-router-many-pages` completes (one run, 34.3 s compiled / 59 s wall; not a benchmark). ### Benchmarks `bench/nested-deps-app-router-many-pages`, cold `next build` (Turbopack), `taskset` CPUs = `TURBO_TASKS_AVAILABLE_PARALLELISM` workers. Current head (`4d89cde5`) vs its base canary (`b24905530`), both fresh release builds of the native bindings with the same JS. Per CPU count: 1 warm-up per arm, then 4 interleaved untraced rounds per arm (medians): | CPUs = workers | canary wall | this PR wall | Δ wall | canary "Compiled in" | this PR "Compiled in" | Δ compiled | paired per-round wall Δ | |---:|---:|---:|---:|---:|---:|---:|---| | 8 | 93.5 s | 93.0 s | −0.5% | 52.5 s | 51.5 s | −1.9% | −3.3 … +8.1% | | 16 | 73.6 s | 72.5 s | −1.4% | 42.5 s | 41.5 s | −2.4% | −5.9 … +4.4% | | 32 | 69.5 s | 59.2 s | **−14.8%** | 45.5 s | 34.0 s | **−25.2%** | −25.4 … −5.3% (all 4 faster) | `whole app module graph` span, from 2 light-traced runs per arm (`NEXT_TURBOPACK_TRACING=next_api=info`); busy cores are system-wide `/proc/stat` over the span, as a share of the pinned CPUs: | CPUs | canary span | this PR span | Δ | busy cores, canary | busy cores, this PR | |---:|---:|---:|---:|---:|---:| | 8 | 9.56, 9.16 s | 8.91, 8.81 s | −5% | 6.6 of 8 | 6.7 of 8 | | 16 | 7.67, 6.99 s | 5.76, 5.69 s | −22% | ~11 of 16 | ~12 of 16 | | 32 | 10.36, 12.35 s | 5.60, 5.94 s | **−49%** | ~12 of 32 | ~23 of 32 | At 8 and 16 CPUs the wall-time differences are within run-to-run noise. At 32 CPUs, canary keeps only ~12 cores busy during the module graph; this PR roughly doubles that and halves the phase. Earlier passes, for history: the first version of this PR measured 77.5 s → 57.9 s wall at 32 CPUs (older base). The first simplification pass showed no measurable difference to the version before it (8 rounds at 32 CPUs: 63.4 s → 62.2 s median wall, paired per-round median +0.3%). Review-round microbenchmarks (scratch harnesses, not committed): - `ConcurrentQueue` vs `crossbeam_queue::SegQueue`: SegQueue was 16–30% slower at 16/32 threads with split producers/consumers and 3–18% slower with mixed push/pop at 32 threads (two runs), so `ConcurrentQueue` stays and no full build benchmark was run. - `fetch_add(0)` RMW vs fence + `load` on the saturated path: 1.97 s vs 1.94 s median for 4.8M schedules, within noise. No speedup is claimed. - Layout: `PriorityRunner` is already 128-byte aligned (via `ConcurrentQueue`'s `CachePadded`), so extra padding isn't needed. Caveats: single VM, one session, 4 untraced and 2 traced rounds per arm, no statistical test; the CPU counts ran one after another, so cross-CPU-count comparisons can include drift. The AtomicBool's own effect isn't isolated. <!-- NEXT_JS_LLM --> <!-- fleet d8fee2f9-18db-4936-b59a-f8417b8f4bbc --> --------- Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by: Luke Sandberg <lukesandberg@users.noreply.github.com> Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com> Co-authored-by: Vercel Fleet <fleet@vercel.com>