perf: Make turbo setup faster on cold and reused runs (#14942)
### Description
`turbo setup` was slow on macOS in two places. A cold install spent most
of its time flushing files to disk. A run that reuses every tool waited
almost a second before printing anything.
```diff
turbo setup
- build 8 download clients, each loading the OS trust store
+ build one download client at the first download
- prepare node, then pnpm, then go, ...
+ prepare node, pnpm, go, ... concurrently
for each missing tool: install_tree
- clone the extracted tree into staging, then delete the extracted tree
+ rename the extracted tree into staging
make_read_only, tree_hash
- sync_tree: F_FULLFSYNC per file and directory
+ sync_tree: fsync per file and directory, then one F_FULLFSYNC
```
- **Flush once per tree.** On macOS, Rust's `File::sync_all` is
`F_FULLFSYNC`, which empties the drive's whole write cache and costs
about 6 ms. `sync_tree` called it for every file. It now calls a plain
`fsync` per entry and one `F_FULLFSYNC` at the end, which commits
everything the earlier calls wrote. The tree is as durable as before
when `publish` renames it into the store. Other platforms are unchanged.
- **Concurrent downloads.** Each adapter's `prepare` now runs in one
`tokio::try_join!`. The repository checks between downloads only avoided
wasted traffic. Setup still runs the same checks before the first
download and before it stages anything.
- **Move instead of copy.** `move_tree` renames the extracted root over
the empty staging directory. If the system temp directory and the store
are on different filesystems, it falls back to a copy. Windows always
copies, so files inherit the store's protected DACL. A second stage of
the same preparation now fails instead of sealing an empty tree.
- **One lazy download client.** Every `DownloadClient::new` built its
own `reqwest` client, and each build read the OS trust store (about 80
ms on macOS). These clients now share one process-wide client, built at
the first request. A run that downloads nothing never reads the trust
store. `with_http_builder` still builds eagerly, so injected trust fails
at construction.
Three tests change. `mirror_tests` and `force_tests` asserted that a
Node failure stops pnpm requests, which held only while downloads ran in
order. The recovery test in `node_provision` now prepares again before
it retries, as a real second run does. It also asserts that staging a
spent preparation fails and publishes nothing.
### Testing Instructions
I measured on an M-series Mac with APFS, using release builds and a
fresh `TURBO_TOOL_STORE` for each cold run. Runs alternated between the
old and new builds.
- **Cold install** in a repository with Go, Python, uv, Node, and pnpm
(19,512 tree files). The median of 3 runs went from 78.7 s to 25.9 s.
Downloads are now bound by link bandwidth (17.6 MB/s here, about 170 MB
of artifacts), so concurrent downloads did not change the number on this
link.
- **Reused tools** in this repository. The first output line went from
0.93 s to 0.23 s, and the full run went from 1.55 s to 0.86 s (5 runs
each). Most of what remains is `cargo fetch --locked`, which setup
reruns every time.
- `turbo setup --check` passes on trees that the new build installed,
and `go`, `python3`, `node`, `uv`, and `pnpm` run from the store's
environment `bin`.
To try it:
1. Run `cargo build --release -p turbo`.
2. In a repository with `futureFlags.experimentalSetup`, run
`TURBO_TOOL_STORE=$(mktemp -d) ./target/release/turbo setup` for a cold
install.
3. Run `./target/release/turbo setup` twice. The second run reuses every
tool.