turbo
7eb014fe - perf: Make turbo setup faster on cold and reused runs (#14942)

Commit
17 days ago
perf: Make turbo setup faster on cold and reused runs (#14942) ### Description `turbo setup` was slow on macOS in two places. A cold install spent most of its time flushing files to disk. A run that reuses every tool waited almost a second before printing anything. ```diff turbo setup - build 8 download clients, each loading the OS trust store + build one download client at the first download - prepare node, then pnpm, then go, ... + prepare node, pnpm, go, ... concurrently for each missing tool: install_tree - clone the extracted tree into staging, then delete the extracted tree + rename the extracted tree into staging make_read_only, tree_hash - sync_tree: F_FULLFSYNC per file and directory + sync_tree: fsync per file and directory, then one F_FULLFSYNC ``` - **Flush once per tree.** On macOS, Rust's `File::sync_all` is `F_FULLFSYNC`, which empties the drive's whole write cache and costs about 6 ms. `sync_tree` called it for every file. It now calls a plain `fsync` per entry and one `F_FULLFSYNC` at the end, which commits everything the earlier calls wrote. The tree is as durable as before when `publish` renames it into the store. Other platforms are unchanged. - **Concurrent downloads.** Each adapter's `prepare` now runs in one `tokio::try_join!`. The repository checks between downloads only avoided wasted traffic. Setup still runs the same checks before the first download and before it stages anything. - **Move instead of copy.** `move_tree` renames the extracted root over the empty staging directory. If the system temp directory and the store are on different filesystems, it falls back to a copy. Windows always copies, so files inherit the store's protected DACL. A second stage of the same preparation now fails instead of sealing an empty tree. - **One lazy download client.** Every `DownloadClient::new` built its own `reqwest` client, and each build read the OS trust store (about 80 ms on macOS). These clients now share one process-wide client, built at the first request. A run that downloads nothing never reads the trust store. `with_http_builder` still builds eagerly, so injected trust fails at construction. Three tests change. `mirror_tests` and `force_tests` asserted that a Node failure stops pnpm requests, which held only while downloads ran in order. The recovery test in `node_provision` now prepares again before it retries, as a real second run does. It also asserts that staging a spent preparation fails and publishes nothing. ### Testing Instructions I measured on an M-series Mac with APFS, using release builds and a fresh `TURBO_TOOL_STORE` for each cold run. Runs alternated between the old and new builds. - **Cold install** in a repository with Go, Python, uv, Node, and pnpm (19,512 tree files). The median of 3 runs went from 78.7 s to 25.9 s. Downloads are now bound by link bandwidth (17.6 MB/s here, about 170 MB of artifacts), so concurrent downloads did not change the number on this link. - **Reused tools** in this repository. The first output line went from 0.93 s to 0.23 s, and the full run went from 1.55 s to 0.86 s (5 runs each). Most of what remains is `cargo fetch --locked`, which setup reruns every time. - `turbo setup --check` passes on trees that the new build installed, and `go`, `python3`, `node`, `uv`, and `pnpm` run from the store's environment `bin`. To try it: 1. Run `cargo build --release -p turbo`. 2. In a repository with `futureFlags.experimentalSetup`, run `TURBO_TOOL_STORE=$(mktemp -d) ./target/release/turbo setup` for a cold install. 3. Run `./target/release/turbo setup` twice. The second run reuses every tool.
Author
Parents
Loading