turbo
ecd58f44 - fix: Make the `turbo trace` graph directory follow `cacheMaxAge` and `cacheMaxSize` (#15051)

Commit
1 day ago
fix: Make the `turbo trace` graph directory follow `cacheMaxAge` and `cacheMaxSize` (#15051) `turbo trace` keeps its whole-repo graph in `.turbo/trace`, and nothing ever cleaned it up: superseded shard versions sat in the store (reclaiming them was left to test code, and the pack only compacted itself once its log passed 32 MB), which is how a branch switch grew the directory from 27 MB to 50 MB. With `cacheMaxAge` or `cacheMaxSize` set, the end of a `turbo trace` command now cleans the graph directory the way a `turbo run` cleans `.turbo/cache`. - Runs `gc` first: every store file the current `graph.json` does not reference goes (superseded shard versions, a log that grew, walked-file lists no snapshot names), so the shards the next command opens are never eviction candidates, whatever their age. - Then applies the two limits to the rest: the graph directory's caches (`hashes`, `manifests`, `walk`, `obs`, `composed`, `tables`). A file last written longer ago than `cacheMaxAge` goes first, then the least recently used ones until the graph directory is under `cacheMaxSize`. - Skips the store rewrite when the store already holds exactly the snapshot, so a graph directory with nothing to reclaim costs nothing. - The graph's caches follow the limits wherever the host keeps them, including the per-machine directory in the user cache dir that `turbo trace` uses. - Never follows a symbolic link: the graph directory, anything above it inside the repository, the store and each cache directory are checked before anything is written or deleted through them, and a link is refused (`Error::Linked`). `gc` deletes only the files a build writes (`is_store_file`), so a repository can't point cleanup at files outside the checkout. - Knows every cache file by name: the code that writes each cache and the eviction share one `cache_file` constant per file, so `CACHE_FILES` covers the cross-file binding caches (`xfile-facts`, `xfile-trace`) and the derived layers too, and a test fails if a command writes a cache file eviction doesn't know about. - Reads the issue's "run `gc` first, then apply age and size limits to the rest" literally: after `gc` the store holds only shards `graph.json` references, so the limits are applied to what a snapshot does not need, never to its shards. The limits are the existing `cacheMaxAge` and `cacheMaxSize` values, and `.turbo/cache` and `.turbo/trace` keep separate budgets, so setting them for tasks does not shrink the graph store and vice versa. Nothing changes without those options set, which also keeps the reuse of superseded shards across edits for anyone who has not opted in; a snapshot larger than `cacheMaxSize` on its own keeps its store and loses only its caches. ### Validation - `cargo test -p turborepo-trace-store` — all targets pass, including the new `eviction_keeps_the_snapshot_and_trims_the_rest`. - `cargo clippy -p turborepo-trace-store -p turborepo-trace -p turborepo-cli --all-targets -- -D warnings`. - `cargo test -p turborepo-trace --lib`, `cargo test -p turborepo-trace --test location`, `cargo test -p turborepo-cli --lib commands::trace`. - `cargo fmt --check`. - Not exercised end to end: no run of the binary against a real repository with the options set. Fixes [TURBO-6555](https://linear.app/vercel/issue/TURBO-6555/make-turbotrace-follow-cachemaxage-and-cachemaxsize) <details><summary>Where the growth came from, and why there is no per-shard last-use table</summary> The store is one `pack` plus an append log. A build appends the shards it wrote; appending a re-keyed package leaves the previous version of that shard in the log, and a load only folds the log into the pack once the log passes 32 MB (`LOG_SLACK` in `compose.rs`) or a quarter of the pack. A branch switch re-keys every package, so the log gains a second copy of the whole snapshot and stops just short of the threshold, which is the 27 MB to 50 MB measurement. `gc` rewrites the pack with exactly the shards of the current `graph.json` and drops the log, so running it periodically is what bounds the directory. There is no last-use table per shard. The store is a pack, not a file per entry, and after `gc` the only shards left are the ones `graph.json` references, which the limits may not remove; a table recording last use would name exactly the keys that must stay. What can be aged or sized is the set of cache files, each of which is a function of the tree or of this machine's file stamps and is recomputed by the next build, so eviction costs a slower build and changes no result. A cache file's mtime is its last write, which is the closest thing to last use a whole-file cache has. Files: - `crates/turborepo-trace-store/src/lib.rs`: `evict` and `EvictReport` — `gc` first, then age then size over the graph directory's caches, and `store_is_current` so a store that already matches its manifest is not rewritten. - `crates/turborepo-trace/src/graph.rs`: `evict_graph`, which resolves `<root>/.turbo/trace` and cannot fail a command. - `crates/turborepo-cli/src/commands/trace.rs`: after the command's work (and the graph's transport), runs the cleanup when `cacheMaxAge` or `cacheMaxSize` is set. - `crates/turborepo-trace-store/src/lib.rs` and `src/artifact.rs`: `real_dir` and `not_link` refuse a symbolic link at the graph directory, above it, at the store or at a cache directory, and `is_store_file` limits deletion to files a build writes. - `crates/turborepo-trace-store/tests/untrusted_store.rs`: a symlinked store is refused, a symlinked cache is refused, and `gc` leaves a link and anything else a build didn't write alone. - `crates/turborepo-trace-store/src/lib.rs` (`cache_file`), with `build.rs`, `compose/*`, `walk.rs`, `turborepo-trace/src/names.rs` and `xfile.rs` switched to it: one constant per cache file name, which `CACHE_FILES` is built from. - `crates/turborepo-trace/tests/cache_files.rs`: runs commands, including cross-file binding, and fails if the cache directory holds a file `CACHE_FILES` doesn't list. - `crates/turborepo-trace-store/tests/compose.rs`: one build, one edit that re-keys a shard, then eviction: the superseded shard version and a stale cache file go, a cache file this build wrote stays until the size limit removes it, and the graph exports the same before and after. </details> --------- Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
Author
Parents
Loading