next.js

8d1f4ed9 - turbo-tasks-backend: release the owner task before the reader side of keyed refinement (#101732)

turbo-tasks-backend: release the owner task before the reader side of keyed refinement (#101732) ## What? Release the keyed cell owner's task lock before updating the reader side of a provisional dependency refinement. This is a minimal port onto canary's flat `EdgeSet` storage from #101489. It remains based directly on `canary`; #101734 builds on it. #101736 is now independent. ## Why? Many readers can refine thousands of key dependencies on the same owner. Holding the owner while updating both sides serializes work that only needs the already-locked reader. Shortening that critical section improves initial bulk keyed registration. Canary already skips redundant owner-edge insertion (#101489), superseding #101733's implementation. Its two re-execution/edge-count tests are retained here. ## How? - Inflate the reader's current forward set before repeated lookups. - With the reader locked, the owner pass checks both outdated and current forward membership to decide which reverse edges are new. - Release the owner, then update the reader's current/outdated sets. The reader stays locked throughout, so an updater that sees the new reverse edges must wait for matching reader state. - The backend accepts a cloneable iterator of **distinct** key hashes. The dyn-compatible manager accepts `&FxHashSet<u64>`, which the real caller already owns. Each lock phase iterates the same keys, without a buffer. - No broader lock-order or `task_pair` redesign is included. The old owner-membership debug assertion is not carried over: with a large flat owner `EdgeSet`, it would add a linear scan per re-read key in debug builds. Exact reverse-edge counts are covered by the retained tests. ## Benchmarks Release `keyed_deps`, N=1000 × M=10,000, shared layout, 3 interleaved blocks at each thread count. Measured base: `cad154f1` (includes #101489). Later rebasing to `dcb536d0` did not change the affected backend/task crates; gates were rerun. Ratios are median paired speedups [min–max], not ratios of displayed median times. The previous layer is the base for this standalone PR. | Mode | Threads | Step | Base ms | A ms | A vs base | |---|---:|---|---:|---:|---| | keyed_reader | 16 | initial | 929 | 424 | **2.09× [1.90–2.40]** | | keyed_reader | 32 | initial | 928 | 455 | **2.24× [2.04–2.32]** | | keyed_reader | 16 | change_all | 33494 | 34388 | 0.98× [0.80–1.02] | | keyed_reader | 32 | change_all | 33473 | 33545 | 1.00× [0.96–1.17] | | keyed_reader | 16 | change_shift | 14203 | 10980 | 1.17× [1.14–1.30] | | keyed_reader | 32 | change_shift | 14210 | 10992 | 1.29× [1.07–1.35] | | get | 16 | initial | 8776 | 7904 | 1.06× [1.02–1.11] | | get | 32 | initial | 9797 | 9503 | 1.02× [0.98–1.11] | | get | 16 | change_all | 42986 | 40084 | 1.08× [1.01–1.08] | | get | 32 | change_all | 42976 | 40683 | 1.04× [1.02–1.06] | | get | 16 | change_shift | 25598 | 23278 | 1.12× [1.00–1.12] | | get | 32 | change_shift | 25642 | 25455 | 1.01× [0.90–1.07] | Other paired medians (16 / 32 threads): | Mode | change_all (2nd) | change_unread | change_shift (2nd) | |---|---|---|---| | keyed_reader | 1.01× / 1.00× | 1.06× / 0.99× | 1.27× / 1.20× | | get | 1.05× / 1.02× | 1.02× / 1.06× | 1.16× / 1.01× | A does **not** fix the flat-set invalidation scan regression: #101736 addresses that separately. The per-key `get` path still takes per-key locks, so its initial run gains little here. ### Memory Single samples per configuration; allocator-committed bytes per edge, **not live allocations**. After the second `change_shift`: | Layout | Threads | Mode | Base B/edge | A B/edge | |---|---:|---|---:|---:| | shared | 16 | keyed_reader | 95.6 | 95.5 | | shared | 32 | keyed_reader | 122.3 | 118.4 | | shared | 16 | get | 94.8 | 95.2 | | shared | 32 | get | 118.6 | 118.7 | | disjoint | 16 | keyed_reader | 149.9 | 123.1 | | disjoint | 32 | keyed_reader | 156.1 | 149.8 | | disjoint | 16 | get | 136.9 | 137.3 | | disjoint | 32 | get | 144.3 | 147.7 | The large 16-thread disjoint retained-commit difference is not attributed or claimed as a reliable saving. Peak RSS is approximately unchanged (shared keyed_reader 0.77→0.78 GiB at 16 threads and 0.95→0.89 GiB at 32; disjoint ~1.2/1.34 GiB). ## Validation - fmt, clippy `-D warnings`, including `task_dirty_cause`. - 625 tests pass across `turbo-tasks` and `turbo-tasks-backend`. - Mutations dropping either registration side or pushing every owner edge are caught by the new tests. - Concurrent registration/change stress, exact reverse-edge counts on repeated executions, and benchmark output/rerun assertions. - Exhaustive concurrent-GC/eviction interleavings and overlapping refinements within one execution are not proven by these tests. No real-build/persistent-cache measurement is claimed. <!-- NEXT_JS_LLM --> <!-- fleet 86487b82-3b2f-4464-84da-b3be3db57b3a --> Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>