turbo-tasks: re-check job counters while stopping (#102142)
### What?
`TurboTasks::stop_and_wait` (and `wait_background_done`) now wait until
the foreground/background job counters actually read 0. Before, a single
wake-up was enough to proceed. This fixes a rare abort when turbo-tasks
shuts down:
```
task TaskId 1073741954 (task, MustExist): task is missing in memory or persistent storage — a stale reference to an already-collected or never-created task
```
This was seen in `turbo-tasks-backend/tests/collectibles_take_batch.rs`
on CI.
### Why?
`stop_and_wait` registered a listener on `event_foreground_done`, saw a
non-zero counter, awaited **one** notification, and then called
`backend.stop()`. `finish_foreground_job` sends that notification only
*after* it decremented the counter (and ran `idle_start` and the stats
code). So the notification for an **earlier** drop to 0 can arrive after
new jobs began.
In the test harness, the sequence is:
1. Run N's `run_once` Once task finishes, and the counter goes 1 → 0.
Its thread is delayed before notifying.
2. Run N+1 starts (0 → 1). Its Once task sends its result through
`tx.send`, so the harness calls `stop_and_wait`, but the task is still
finishing (`Completion::cell()`, `task_execution_completed`).
3. The delayed notification from step 1 wakes `stop_and_wait` with the
counter still at 1. `backend.stop()` drops storage under the running
Once task, which then can't find its own transient task → abort.
Evidence:
- Counter-transition traces from natural failures show `fg check = 1` →
`woken, fg=1` with no 1 → 0 transition in between, followed by
`backend.stop` with one job in flight. The aborting task is the last
run's Once task, with the same TaskId as on CI.
- With a timing injection that forces this interleaving, current canary
hits the early stop in 10/10 runs of `turbo-tasks-backend --test
collectibles`. So this is pre-existing and not specific to any open PR.
- Bisecting with that injection: the early stop first appears with
#91790, which added `snapshot_and_evict` between the harness's in-memory
runs. That likely lets the counter reach 0 between runs. The one-shot
wait itself is older; the panic message changed over time (#93227,
#99249).
### How?
A private `wait_until_zero(counter, event, note)` helper loops: register
a listener, check the counter, return if it is 0, otherwise await and
check again. `stop_and_wait` uses it for the foreground counter and then
for the background counter (via `wait_background_done`).
### Verification
- New unit tests in `manager.rs`: a stale notification while the counter
is 1 must not end the wait, and the wait ends immediately at 0.
Temporarily swapping the old one-shot logic back in makes
`test_wait_until_zero_ignores_stale_notification` fail. The tests also
pass with `--features hanging_detection`.
- Forced interleaving (local, not committed): `turbo-tasks-backend
--test collectibles` went from early stop + abort in 10/10 runs to 0/10,
with all tests passing.
- Natural stress (local, 16 parallel copies, `release-with-assertions`):
`collectibles_take_batch` aborted in 11 of 150,000 runs on a
scheduler-changing PR commit (with counter tracing) and in 0 of 150,000
with this fix. The failure is rare and timing-dependent, so this is
supporting evidence only.
- `cargo test -p turbo-tasks -p turbo-tasks-backend --profile
release-with-assertions` (existing tests, unchanged behavior)
- `cargo clippy -p turbo-tasks --all-targets`
- Not run locally: the full workspace `cargo nextest` (CI covers it).
<!-- NEXT_JS_LLM -->
<!-- fleet bb1e95d3-28b6-439b-8d55-374604395c38 -->
Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>