feat: Add experimental `turbo trace` (#14992)
# feat: Add experimental turbo trace
## Description
This PR adds `turbo trace`, an experimental command that selects the
**test files** a change can affect.
`turbo run --affected` and `turbo query affected` select packages and
tasks; `turbo trace affected` goes one level further and names the test
files. It follows each test's imports and other references through the
repository: re-exports, jest, vitest and Next.js configs, tsconfig
paths, lockfile changes, scripts, and files read by literal path. It
reports a test when a changed file is among that test's transitive
inputs.
The command is **off unless `TURBO_EXPERIMENTAL_TRACE=1` is set**, and
it is hidden from `turbo --help`.
- Without the variable, `turbo trace …` parses exactly as it does on
`main`, as `turbo run trace …`. That holds for the shim and managed
setup too, so a repository task named `trace` keeps working.
- Nothing else in `turbo` changes behaviour. The variable is not a
task-hash input, and `turbo run` does not touch the new code.
```bash
TURBO_EXPERIMENTAL_TRACE=1 turbo trace affected # base: merge base with main → working tree
TURBO_EXPERIMENTAL_TRACE=1 turbo trace affected --base origin/main --json
TURBO_EXPERIMENTAL_TRACE=1 turbo trace why packages/math/test/add.test.js packages/math/src/add.js
TURBO_EXPERIMENTAL_TRACE=1 turbo trace lint
```
### Design
- **Soundness first.** A test is left out only when no changed file can
reach it. Whatever the analysis can't resolve (a computed import, an
unreadable config, a test it can't place) is reported, not silently
dropped:
- tests it can't place are listed with the reason `UnknownTest`. Without
a jest or vitest config, the runners' default patterns decide what a
test is;
- tasks whose runner it can't map are listed under `uncoveredTasks`;
- `--exit-code` counts both;
- an SCM error, or a root turbo.json other than the default, selects
every test;
- `--strict` also counts the stated assumptions;
- `--explain` prints the chain from a test to a changed file.
- **Built on turbo's own model.** The command builds a `Run` the way
`turbo boundaries` and `turbo query affected` do. It builds the task
graph only when `--filter` needs it, as `turbo ls` does. It takes:
- the workspace's JS packages, from `PackageGraph`;
- the changed files, from `SCM::changed_files`. These are the same files
`turbo run --affected` sees, with turbo's base resolution, ref
validation and `TURBO_SCM_BASE`/`TURBO_SCM_HEAD`. They are prefetched
while the package graph is built (a new
`RunBuilder::prefetch_changed_files`, which does not narrow
`filtered_pkgs`). A small layer on top adds the status (added, modified,
deleted) that the diff rules need.
Flags follow `turbo query affected`: `--base`, `--head`, `--exit-code`,
plus `--filter`. The JSON uses the same list shape as `affectedTasks`:
`{ affectedTests: { items: [{ path, package, task, reason: { __typename
… } }], length }, uncoveredTasks }`.
- **Storage and caching.**
- The graph lives in `.turbo/trace/` and is refreshed against the tree
on each run.
- Caches keyed by this machine's file timestamps live in the user cache
directory, never in the repository, so a `.turbo` restored by
`actions/cache` into another checkout is safe.
- Everything in `.turbo/trace`, and every cache entry, is treated as
untrusted input. File names are checked against their canonical form
before any read or delete. Every archive is structurally validated on
load, in release builds too.
- In CI the graph travels through Turborepo's own cache (`AsyncCache`),
so local and remote caching, signing, `--cache-dir` and `--cache` all
apply.
- The key is the commit SHA, the `turbo` version, the OS and the
architecture.
- A run looks up HEAD, then the merge base, then up to 20 first-parent
ancestors.
- A restored graph is refreshed against the tree before use.
- **Debug switches.** The analysis has internal kill switches for
diagnosis. They are compiled out of release builds, behind the
off-by-default cargo feature `debug-knobs` of `turborepo-trace-core`,
which lists them. `turbo`'s own integration tests run the shipped
configuration, without them.
- **Errors, lints and tracing.**
- One `thiserror` + `miette::Diagnostic` error type at the library
boundary.
- Every `unwrap`/`expect` outside tests has a narrow `#[allow]` with its
invariant written next to it, and there are no blanket allows.
- The entry points and the cache transport have `tracing` spans.
### Layout
Ten library crates plus a dev-only test kit, all under
`crates/turborepo-trace*`. Only `turborepo-trace` depends on other turbo
crates.
| Crate | What it does |
|---|---|
| `turborepo-trace` | command logic for `affected`/`why`/`lint`, and the
adapters onto `Run`, `PackageGraph`, `SCM` and the cache |
| `turborepo-trace-core` | data model, scheduler, file-system cache,
path resolution, partial evaluation |
| `turborepo-trace-lang-js` | JS/TS analysis (oxc parser,
`oxc_resolver`) |
| `turborepo-trace-lang-go`, `-lang-rust` | Go and Rust analysis (cargo
features `go` and `rust`, on by default, about 1.1 MB) |
| `turborepo-trace-frameworks` | jest, vitest, Next.js and tsconfig
adapters |
| `turborepo-trace-tasks` | recognizers for package scripts and other
task sources |
| `turborepo-trace-store` | the sharded graph and incremental refresh |
| `turborepo-trace-graph` | the graph query engine |
| `turborepo-trace-lockfile` | per-importer lockfile diffs (JS, Go and
Cargo lockfiles) |
| `turborepo-trace-testkit` | test-only: fixtures and a stub language
(`publish = false`) |
Outside those crates, the changes are small:
- the `Trace` command and its gate (`turborepo-cli`: `args.rs`,
`commands/trace.rs`, `shim.rs`);
- `RunBuilder::prefetch_changed_files`;
- `GitRepo::resolve_base` and `validate_git_ref` made public;
- `batch_exists` on the cache multiplexer, and a `cache_dir()` helper in
`turborepo-dirs`;
- an oxc bump for `turbo-trace` and boundaries;
- the docs page `reference/trace.mdx` and the env var row;
- a nextest override that runs only the gate test on Windows.
### How to review
- **Read line by line, about 4k lines.** The changes outside
`crates/turborepo-trace*`, plus
`crates/turborepo-trace/src/{api,adapter,transport}.rs` and
`crates/turborepo-cli/src/commands/trace.rs`.
- **Review by tests and behaviour.** The layer commits that add
`turborepo-trace-core` through `-store` import an analysis engine that
ran as a standalone tool before this PR. They are covered by about 2,200
tests, by integration tests through `turbo`, and by a corpus comparison:
on a few hundred recorded CI changes from two large monorepos, `turbo
trace affected` selected the same tests as the standalone tool, plus the
test files that jest's default patterns match. In both monorepos, every
test that failed in CI for a reason other than flakiness was selected.
The commits build and pass in order:
1. `chore: Update oxc crates to 0.156.0`: turbo's existing oxc users (4
files).
2. The five layer commits `feat: Add turborepo-trace-*`: the imported
analysis crates.
3. `feat: Add the turborepo-trace library and its turbo adapters`.
4. `feat: Add experimental turbo trace`: the command, gate, storage
under `.turbo/trace`, transport through turbo's cache, docs and
integration tests.
5. `test:` and `build:` commits: the Windows nextest override, and
`opt-level = "s"` for some trace crates in `release-turborepo`.
6. `fix:`, `refactor:`, `perf:` and `chore:` commits that harden the
command and fit it to turbo:
- untrusted store contents;
- managed setup for gate-off `turbo trace`;
- jest/vitest default patterns;
- changed files from turbo's SCM;
- no lint baseline file;
- unreachable code removed;
- the root turbo.json;
- no task graph unless needed;
- tracing spans.
### Deliberately duplicated, with follow-ups
- **Resolver:** `oxc_resolver` 11 sits beside turbo's `unrs_resolver`,
and the selection's resolution semantics and tests depend on it.
Follow-up: one resolver, for example by moving `turbo-trace` and
boundaries to `oxc_resolver`.
- **Lockfile parser:** `turborepo-trace-lockfile` sits beside
`turborepo-lockfiles`; ours computes the per-importer dependency diffs
that selection depends on. Follow-up: add those diffs to
`turborepo-lockfiles`.
- **turbo.json reading:** `turbo.json` is read again for the graph's
task model. A parity test checks `dependsOn` and `outputs` against
`turbo run --dry=json` over 43 fixtures and 134 tasks. Follow-up: take
the definitions from `TurboJsonLoader`.
- **Import tracers:** `crates/turbo-trace`, the import tracer behind
`boundaries` and `query file`, is untouched; its README now says so.
Follow-up: move them onto `turborepo-trace-lang-js`.
### Known gaps
- An untracked test file hidden by a `.ignore` file counts as a change
but is not discovered as a test, because the graph's walk honours
`.ignore`.
- The task definitions of turbo's native Cargo and uv packages are not
read for the graph's task model.
- Go and Rust analysis is validated on fixtures and on one monorepo's
history, less than JS/TS. A TS-only change can still select some Go
tests that read files by computed paths: this over-selects, and never
drops a test.
- Windows: it compiles, and at run time it exits with "not supported on
Windows yet".
### Not in this PR
- Test-level selection inside `turbo run test --affected`. The plan: an
env var for the skip list, plus an optional digest in `TaskHashable`
written only when set, so full runs keep today's hash.
- An `affectedTests` GraphQL field and `turbo query affected --tests`.
- A `trace` key in `turbo.json`.
- Python and shell analysis.
- Further follow-ups:
- `tracing` spans in place of the engine's own timing;
- a context object in place of two process-wide switches;
- one glob implementation for task globs;
- subcommand telemetry;
- structured errors inside the engine;
- splitting the largest source files.
### Questions for maintainers
1. The graph is stored in the task cache under its own key. Is a
non-task artifact there acceptable (key namespace, analytics, eviction),
or should it get a prefix or its own endpoint?
2. Should the JSON use the `{"data": …}` envelope of `turbo query` now,
to ease graduation?
3. Is `turbo trace` the right name, next to the global `--trace` flag
and `crates/turbo-trace`?
4. Would you take a status-carrying `changed_files`, and caller-supplied
untracked files, in `turborepo-scm`?
5. Is the `turbo trace: …` stderr style acceptable, or should it go
through `turborepo_ui`?
6. `Run` already builds an untracked-file index in the background. May
trace read it?
### Cost
- **Size:** the stripped `turbo` binary (`release-turborepo`, Linux
x86_64) grows from 50.9 MB to **68.6 MB (+17.7 MB, +35%)**. Compressed
with xz it is 17.8 MB. `opt-level = "s"` is set for the trace crates
that are off the hot path.
- **Build time:** a cold release build of `turbo` is unchanged within
noise.
- **Lines:** about 245k lines of Rust in `src`, about 70k lines of
tests, and about 12 MB of test fixtures.
- **`turbo run`:** unchanged with the gate off. `--dry=json` timings on
two large monorepos and on a generated 1,500-package fixture are within
noise of `main`, and the dry-run JSON is identical.
- **`turbo trace affected`, warm:** a no-op, an edit or a `--changed`
run takes 0.43–1.0× the time of the same analysis as a standalone
binary, not counting `RunBuilder`. Building the `Run` (package graph,
lockfile) is a fixed cost on top, the same one `turbo query` pays: about
a second on a very large workspace, a few hundred milliseconds on a
mid-sized one.
- **Cold, and restore through the cache:** cold runs take 0.90–1.04× the
time of the standalone binary. A graph restored through turbo's cache
takes 0.80–0.95×.
- **Peak memory:** about 20–35% above the standalone, roughly
`RunBuilder`'s own peak.
## Testing Instructions
- `cargo test -p turbo --test trace_test`: 52 integration tests over the
`trace_jest`, `trace_vitest`, `trace_polyglot` and `trace_go_climb`
fixtures, in the shipped configuration. They cover:
- **the gate:** off, unset, `0` and `false`, a byte-identical tree, the
`turbo run trace` fallback, and hidden help;
- **selection:** changes, adds, deletes, renames and directory changes,
a base that has moved on, and jest default test files without a config;
- **other languages:** a workspace with vitest, a Go module and a Rust
crate that share one data file. A Go, Rust or TS change selects only
that language's tests, and a change to the shared file selects the tests
of all three that read it;
- **output and commands:** `--json` stability, `--exit-code`, `why`, and
`lint`, which reads no baseline file;
- **the graph's storage:**
- restored through a local cache directory into a fresh clone;
- `.turbo` copied into a second checkout;
- a pull-request run with the GitHub Actions base.
- `cargo test -p turbo --test trace_task_definitions_test`: the trace's
task model against `turbo run --dry=json`.
- `cargo test -p turbo --test managed_run_pin_test`: gate-off `turbo
trace` gets the same managed-setup check as `turbo run`, and gate-on
`turbo trace` is routed like `turbo query`.
- `cargo test -p turborepo-cli`: help snapshots and argument parsing.
- `cargo test -p 'turborepo-trace*'`: the analysis crates' tests
(2,215). They include crafted `graph.json` names and forged shards with
valid checksums.
- Manual:
```bash
cd turborepo-tests/integration/fixtures/trace_jest
git init -q && git add -A && git commit -qm base
echo '// edit' >> packages/math/src/add.js
TURBO_EXPERIMENTAL_TRACE=1 turbo trace affected --explain
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Anthony Shew <anthony.shew@vercel.com>