next.js
6ffff505 - turbo-tasks: re-check job counters while stopping (#102142)

Commit
1 day ago
turbo-tasks: re-check job counters while stopping (#102142) ### What? `TurboTasks::stop_and_wait` (and `wait_background_done`) now wait until the foreground/background job counters actually read 0. Before, a single wake-up was enough to proceed. This fixes a rare abort when turbo-tasks shuts down: ``` task TaskId 1073741954 (task, MustExist): task is missing in memory or persistent storage — a stale reference to an already-collected or never-created task ``` This was seen in `turbo-tasks-backend/tests/collectibles_take_batch.rs` on CI. ### Why? `stop_and_wait` registered a listener on `event_foreground_done`, saw a non-zero counter, awaited **one** notification, and then called `backend.stop()`. `finish_foreground_job` sends that notification only *after* it decremented the counter (and ran `idle_start` and the stats code). So the notification for an **earlier** drop to 0 can arrive after new jobs began. In the test harness, the sequence is: 1. Run N's `run_once` Once task finishes, and the counter goes 1 → 0. Its thread is delayed before notifying. 2. Run N+1 starts (0 → 1). Its Once task sends its result through `tx.send`, so the harness calls `stop_and_wait`, but the task is still finishing (`Completion::cell()`, `task_execution_completed`). 3. The delayed notification from step 1 wakes `stop_and_wait` with the counter still at 1. `backend.stop()` drops storage under the running Once task, which then can't find its own transient task → abort. Evidence: - Counter-transition traces from natural failures show `fg check = 1` → `woken, fg=1` with no 1 → 0 transition in between, followed by `backend.stop` with one job in flight. The aborting task is the last run's Once task, with the same TaskId as on CI. - With a timing injection that forces this interleaving, current canary hits the early stop in 10/10 runs of `turbo-tasks-backend --test collectibles`. So this is pre-existing and not specific to any open PR. - Bisecting with that injection: the early stop first appears with #91790, which added `snapshot_and_evict` between the harness's in-memory runs. That likely lets the counter reach 0 between runs. The one-shot wait itself is older; the panic message changed over time (#93227, #99249). ### How? A private `wait_until_zero(counter, event, note)` helper loops: register a listener, check the counter, return if it is 0, otherwise await and check again. `stop_and_wait` uses it for the foreground counter and then for the background counter (via `wait_background_done`). ### Verification - New unit tests in `manager.rs`: a stale notification while the counter is 1 must not end the wait, and the wait ends immediately at 0. Temporarily swapping the old one-shot logic back in makes `test_wait_until_zero_ignores_stale_notification` fail. The tests also pass with `--features hanging_detection`. - Forced interleaving (local, not committed): `turbo-tasks-backend --test collectibles` went from early stop + abort in 10/10 runs to 0/10, with all tests passing. - Natural stress (local, 16 parallel copies, `release-with-assertions`): `collectibles_take_batch` aborted in 11 of 150,000 runs on a scheduler-changing PR commit (with counter tracing) and in 0 of 150,000 with this fix. The failure is rare and timing-dependent, so this is supporting evidence only. - `cargo test -p turbo-tasks -p turbo-tasks-backend --profile release-with-assertions` (existing tests, unchanged behavior) - `cargo clippy -p turbo-tasks --all-targets` - Not run locally: the full workspace `cargo nextest` (CI covers it). <!-- NEXT_JS_LLM --> <!-- fleet bb1e95d3-28b6-439b-8d55-374604395c38 --> Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Author
Parents
Loading