[turbopack] Enable Effects to be evicted (#94173)
### What?
Enable Effects to be evicted.
Currently `EffectInstance` transitively holds a
`ReadRef<PersistentFileContent>` which means eviction cannot not free
the memory for the outputs. Because the `ReadRefs` are captured by the
task producing the `Effect` we cannot just drop them (otherwise we could
need to somehow re-execute the write to recover them). So instead we
need a new 'effect processing phase'
Here is our new state machine:
* Phase 1: `emit` effects.
- At this point we capture the content as a `ResolvedVc` and the hash as
a value.
- In principle this means the hash and content can get out of sync, but
only within an eventual consistency race which callers should be
handling.
* Phase 2: `take_effects`.
- This will call `capture()` on all the effects which will conditionally
read the content cell if the hash doesn't match the `EffectStateStorage`
* Phase 3: `Effects::apply`.
- This will apply the effects for all effects whose hashes don't match
storage.
- At this point we may discover that a hash in storage changed between
`capture` and `apply` which will trigger a special `Retry` error and
`invalidate` `Phase 2`.
To facilitate the `Retry` logic a
`read_strongly_consistent_and_apply_effects` helper function handles it.
Which allows for a simple circuit breaker and some verbose
warnings/errors when we retry too many times.
Specifically:
- Split the `Effect` trait into `Effect` (emit-side, returns a
`Captured` payload) and `CapturedEffect` (apply-side). `
- Switch `Effects` to `cell = "new"`. Every producer re-execution
allocates a fresh cell value; the prior cell (and its captured payload)
drops on overwrite, cascade-releasing upstream
`ReadRef<PersistedFileContent>` strong counts. The captured payload also
drops naturally when the cell is evicted.
- Push the per-key state machine (`Unapplied` / `InProgress` / `Applied
{ value_hash, result }`, `EventGuard` panic recovery, `Event` listener
coordination) down into `EffectStateStorage::run_apply`.
`Effects::apply` becomes a thin aggregator over `dyn_apply()` calls.
- New `ApplyOutcome<E> { Failed(E), Retry }`. `CapturedEffect::apply`
returns this; `Retry` signals "capture elided content but storage
diverged before apply" — `Effects::apply` collects across the batch,
fires the producer's invalidator once, and surfaces
`EffectsError::Retry`.
- New `EffectStateStorage::matches_applied(key, hash)`.
`WriteEffect::capture` / `WriteLinkEffect::capture` consult this and
skip the `ReadRef<PersistedFileContent>` / `ReadRef<LinkContent>`
materialization when storage already holds `Applied { matching, Ok(())
}` — avoiding the disk-read + decompression hit on producer re-runs that
don't actually change output.
- 10 integration tests in
`turbopack/crates/turbo-tasks-backend/tests/effects.rs` cover the new
lifecycle: duplicate apply, sibling stomp re-apply, repeated apply with
unchanged state dedupe, `cell = "new"` producing distinct cells per
producer run, capture-skip on storage match, and the
capture-skip-then-stomp race that exercises the `Retry` path.
### Why?
A memory audit showed `EffectInstance` cells accumulating the majority
of the ram in an eviction session
#### Capture-time storage check (the perf story)
`Effect::capture` may consult `EffectStateStorage::matches_applied` and
elide the content materialization when storage already records the
target hash. `CapturedWriteEffect.content` becomes
`Option<ReadRef<PersistedFileContent>>` — `None` means "capture observed
`Applied { matching }`; no `body` to pass to `run_apply`". The
apply-side state machine then either dedup-hits (storage still matches →
cached `Ok`) or enters the no-body branch and returns
`ApplyOutcome::Retry`.
Reading shared mutable `EffectStateStorage` from inside a turbo-tasks
task is normally suspect (the state isn't a tracked input). It is sound
here because the apply-time re-check fires the producer's invalidator on
mismatch, turning the otherwise-untracked read into an
explicitly-tracked one via the `Retry` pathway. The producer reruns,
`capture` sees the new storage state, materializes content, and the next
`apply` succeeds.
#### `ApplyOutcome::Retry` and the invalidator
`Effects::apply` collects `Retry` signals across the parallel batch via
`AtomicBool`. `Failed(e)` errors fail-fast. After the batch, if any
`Retry` fired (and no `Failed`), the producer's invalidator runs once
and `EffectsError::Retry` propagates. Callers re-read the operation; the
producer reruns; fresh `capture` either materializes content (storage
diverged) or dedup-hits cleanly.
#### `turbo-tasks-fs`
`WriteEffect::capture` / `WriteLinkEffect::capture` call
`matches_applied` first and conditionally `.await?` the content
`ResolvedVc`. `CapturedWriteEffect::apply` /
`CapturedWriteLinkEffect::apply` dispatch through `run_apply(key, hash,
body)`, where `body` is `Some(closure)` iff content is `Some`.
`apply_inner` takes the unwrapped `&ReadRef<…>` as a parameter so the
option-unwrap is at the dispatch boundary (type-enforced).
### Risk: infinite retry loop
**Failure mode.** Two sibling producers A and B contend on the same key
with different hashes (e.g. two routes that both want to write
`chunks/foo.js` with different content). Each `Retry` from A's apply
invalidates A's producer; each `Retry` from B's apply invalidates B's
producer. In the worst case, A and B perpetually re-capture, race to
apply, stomp each other's storage state, and re-trigger Retry —
invalidating themselves on each cycle.
**Why we believe it terminates in practice.** A `Retry` only fires when
(a) capture observed `Applied { matching }` and elided content, and (b)
by apply time storage was stomped to a different hash. Each `Retry`
invalidates only its own producer (no cross-producer invalidation), so
without external input churn the system reaches a fixed point —
whichever producer applies last wins, and subsequent re-runs of the
loser's producer materialize content (because storage no longer matches)
and apply cleanly. The pathological case requires a real, ongoing race
fundamentally triggered by external invalidations (file edits).
<!-- NEXT_JS_LLM_PR -->
