fix: Make the `turbo trace` graph directory follow `cacheMaxAge` and `cacheMaxSize` (#15051)
`turbo trace` keeps its whole-repo graph in `.turbo/trace`, and nothing
ever cleaned it up: superseded shard versions sat in the store
(reclaiming them was left to test code, and the pack only compacted
itself once its log passed 32 MB), which is how a branch switch grew the
directory from 27 MB to 50 MB. With `cacheMaxAge` or `cacheMaxSize` set,
the end of a `turbo trace` command now cleans the graph directory the
way a `turbo run` cleans `.turbo/cache`.
- Runs `gc` first: every store file the current `graph.json` does not
reference goes (superseded shard versions, a log that grew, walked-file
lists no snapshot names), so the shards the next command opens are never
eviction candidates, whatever their age.
- Then applies the two limits to the rest: the graph directory's caches
(`hashes`, `manifests`, `walk`, `obs`, `composed`, `tables`). A file
last written longer ago than `cacheMaxAge` goes first, then the least
recently used ones until the graph directory is under `cacheMaxSize`.
- Skips the store rewrite when the store already holds exactly the
snapshot, so a graph directory with nothing to reclaim costs nothing.
- The graph's caches follow the limits wherever the host keeps them,
including the per-machine directory in the user cache dir that `turbo
trace` uses.
- Never follows a symbolic link: the graph directory, anything above it
inside the repository, the store and each cache directory are checked
before anything is written or deleted through them, and a link is
refused (`Error::Linked`). `gc` deletes only the files a build writes
(`is_store_file`), so a repository can't point cleanup at files outside
the checkout.
- Knows every cache file by name: the code that writes each cache and
the eviction share one `cache_file` constant per file, so `CACHE_FILES`
covers the cross-file binding caches (`xfile-facts`, `xfile-trace`) and
the derived layers too, and a test fails if a command writes a cache
file eviction doesn't know about.
- Reads the issue's "run `gc` first, then apply age and size limits to
the rest" literally: after `gc` the store holds only shards `graph.json`
references, so the limits are applied to what a snapshot does not need,
never to its shards.
The limits are the existing `cacheMaxAge` and `cacheMaxSize` values, and
`.turbo/cache` and `.turbo/trace` keep separate budgets, so setting them
for tasks does not shrink the graph store and vice versa. Nothing
changes without those options set, which also keeps the reuse of
superseded shards across edits for anyone who has not opted in; a
snapshot larger than `cacheMaxSize` on its own keeps its store and loses
only its caches.
### Validation
- `cargo test -p turborepo-trace-store` — all targets pass, including
the new `eviction_keeps_the_snapshot_and_trims_the_rest`.
- `cargo clippy -p turborepo-trace-store -p turborepo-trace -p
turborepo-cli --all-targets -- -D warnings`.
- `cargo test -p turborepo-trace --lib`, `cargo test -p turborepo-trace
--test location`, `cargo test -p turborepo-cli --lib commands::trace`.
- `cargo fmt --check`.
- Not exercised end to end: no run of the binary against a real
repository with the options set.
Fixes
[TURBO-6555](https://linear.app/vercel/issue/TURBO-6555/make-turbotrace-follow-cachemaxage-and-cachemaxsize)
<details><summary>Where the growth came from, and why there is no
per-shard last-use table</summary>
The store is one `pack` plus an append log. A build appends the shards
it wrote; appending a re-keyed package leaves the previous version of
that shard in the log, and a load only folds the log into the pack once
the log passes 32 MB (`LOG_SLACK` in `compose.rs`) or a quarter of the
pack. A branch switch re-keys every package, so the log gains a second
copy of the whole snapshot and stops just short of the threshold, which
is the 27 MB to 50 MB measurement. `gc` rewrites the pack with exactly
the shards of the current `graph.json` and drops the log, so running it
periodically is what bounds the directory.
There is no last-use table per shard. The store is a pack, not a file
per entry, and after `gc` the only shards left are the ones `graph.json`
references, which the limits may not remove; a table recording last use
would name exactly the keys that must stay. What can be aged or sized is
the set of cache files, each of which is a function of the tree or of
this machine's file stamps and is recomputed by the next build, so
eviction costs a slower build and changes no result. A cache file's
mtime is its last write, which is the closest thing to last use a
whole-file cache has.
Files:
- `crates/turborepo-trace-store/src/lib.rs`: `evict` and `EvictReport` —
`gc` first, then age then size over the graph directory's caches, and
`store_is_current` so a store that already matches its manifest is not
rewritten.
- `crates/turborepo-trace/src/graph.rs`: `evict_graph`, which resolves
`<root>/.turbo/trace` and cannot fail a command.
- `crates/turborepo-cli/src/commands/trace.rs`: after the command's work
(and the graph's transport), runs the cleanup when `cacheMaxAge` or
`cacheMaxSize` is set.
- `crates/turborepo-trace-store/src/lib.rs` and `src/artifact.rs`:
`real_dir` and `not_link` refuse a symbolic link at the graph directory,
above it, at the store or at a cache directory, and `is_store_file`
limits deletion to files a build writes.
- `crates/turborepo-trace-store/tests/untrusted_store.rs`: a symlinked
store is refused, a symlinked cache is refused, and `gc` leaves a link
and anything else a build didn't write alone.
- `crates/turborepo-trace-store/src/lib.rs` (`cache_file`), with
`build.rs`, `compose/*`, `walk.rs`, `turborepo-trace/src/names.rs` and
`xfile.rs` switched to it: one constant per cache file name, which
`CACHE_FILES` is built from.
- `crates/turborepo-trace/tests/cache_files.rs`: runs commands,
including cross-file binding, and fails if the cache directory holds a
file `CACHE_FILES` doesn't list.
- `crates/turborepo-trace-store/tests/compose.rs`: one build, one edit
that re-keys a shard, then eviction: the superseded shard version and a
stale cache file go, a cache file this build wrote stays until the size
limit removes it, and the graph exports the same before and after.
</details>
---------
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>