next.js
e186cd70 - Replace jest-worker with custom worker pool (#100297)

Commit
1 day ago
Replace jest-worker with custom worker pool (#100297) ### What? Replaces the vendored `jest-worker` (and the `lib/worker.ts` wrapper around it) with a purpose-built worker pool in `packages/next/src/lib/worker/`. Every Next.js worker uses it: static generation/export, build traces, type checking, the webpack build worker, the minifier, and the dev server's static-paths, dev-validation and `'use cache'` probe workers. The compiled `jest-worker` bundle, its types, its taskfile entry and the devDependency are removed. ### Why? - `jest-worker` is built for Jest's needs. It spawns every worker up front, has no per-worker concurrency or boot throttling, restarts the whole pool when one worker hangs, and hides worker failures behind generic messages (`Call retries were exceeded`, `Jest worker encountered … child process exceptions`). - The `lib/worker.ts` wrapper had to reach into jest-worker's private fields to detect crashes, which only worked for child processes. - A pool we own lets builds spawn only the workers they need, and makes crash handling, flag forwarding and shutdown behave the same for child processes and worker threads. ### How? | File | Purpose | |------|---------| | `worker-pool.ts` | Low-level pool: lazy spawning, per-worker concurrency, boot throttling, task queue, crash recovery, graceful shutdown | | `index.ts` | High-level `Worker`: exposed methods, retries/timeouts, Node.js flag forwarding, debugger re-attachment, color, output forwarding | | `worker-child-common.ts`, `worker-process-child.ts`, `worker-thread-child.ts` | Shared child protocol and the `child_process` / `worker_threads` entry points | | `types.ts` | Message constants and types | | `README.md` | Architecture, message protocol, options | Call sites switch to the new API: `numWorkers` → `maxWorkers`, `.end()` → `await .shutdown()`, and each worker gets a `workerName` used in error messages. ### Differences from the previous implementation **Pool behaviour** | Area | Before (`jest-worker`) | Now | |------|------------------------|-----| | Spawning | All `numWorkers` spawned when the pool is created | Spawned on demand up to `maxWorkers`; at most `maxBootingWorkers` (default `ceil(maxWorkers / 4)`) start at once. Spawned workers stay until shutdown. | | Concurrency | One call per worker | `concurrencyPerWorker` (default 1) | | Hung worker (`timeout`) | The whole pool was replaced, and every in-flight call restarted | Only the hung worker is killed and replaced; `onRestart` fires for the calls it had | | Worker crash | Child processes only: the wrapper logged `Next.js build worker exited with code: …` and called `process.exit` from the parent. Threads weren't detected. | Both modes: the call rejects with `WorkerExitError` (worker name, exit code, signal); `maxRetries` re-dispatches it; other queued calls keep running on a replacement worker | | `worker_threads` arguments | Always JSON round-tripped | Still the default; `sanitizeArgs: false` keeps structured clone (dev validation passes a `SharedArrayBuffer` and typed arrays) | | Graceful shutdown | Workers stopped by signal | `END` → `teardown()` → flush stdout/stderr → `process.exit(0)`, so leftover timers or sockets don't delay shutdown and `process.on('exit')` handlers run. Force-kill after 5 s (60 s with `NEXT_CPU_PROF`) | | Final worker output | Could be cut off at exit | `shutdown()` waits for each worker's stdout/stderr to end (same bound), then detaches and unrefs any pipe still held by a grandchild | **Failure handling** (the same for child processes and worker threads) - A worker that exits (or times out) before it is ready doesn't have calls yet, so they stay queued. After 3 such failures in a row the pool stops replacing it, and the queued calls reject with `WorkerExitError` once no worker is left; a healthy worker still serves them. - A module that throws while loading sends a `LOAD_ERROR` message: its calls reject with `Error when loading worker module: …`, keeping the original error type and stack, and the worker is removed without counting as a crash. - An uncaught async error in a worker thread is printed, and the thread's exit is handled as a crash, so `maxRetries` applies. - A thrown non-`Error` value reaches the caller unchanged (e.g. a thrown string). - A result or error that can't be sent back (BigInt, circular data, function properties), or call arguments that can't be serialized, fail only that call. The worker and its other calls continue, and the queue keeps draining. **Node.js flags and debugging** | Area | Before | Now | |------|--------|-----| | Flag forwarding | `lib/worker.ts` merged CLI flags into `NODE_OPTIONS`, so `NODE_OPTIONS` overrode CLI flags, repeated flags collapsed, and CLI-only flags such as `--prof` crash-looped workers. Raw jest-worker callers had to strip `--inspect` from `NODE_OPTIONS` themselves. | The `Worker` forwards flags the way Node.js does for a child: CLI flags stay CLI flags (`--prof` works), `NODE_OPTIONS` stays `NODE_OPTIONS`. Callers don't pass `NODE_OPTIONS`. | | Debugger flags | Only `--inspect`/`--inspect-brk` stripped from `NODE_OPTIONS`; dev-server worker threads (raw jest-worker) inherited a CLI `--inspect-wait`/`--inspect-brk` and hung | All `--inspect*`/`--debug*` options (including separate values like `--inspect-port 9229`, negations and `--inspect-publish-uid`) are stripped from both, for processes and threads | | Debugger re-attachment (`debuggerPortOffset`) | Address taken from the parsed flags | Taken from Node.js's running inspector (`inspector.url()`): its real host and port, with `--inspect-port` overrides and negations applied, plus `--inspect-publish-uid`. Child processes get it on `parentPort + 1 + offset`; threads never get debugger flags. Only an inspector enabled by a startup flag is passed on; one opened at runtime (`inspector.open()`, SIGUSR1) isn't. The static/export worker offset is `0` (replaces `getNextBuildDebuggerPortOffset`). | | `next dev` debugger | `--inspect`/`--inspect-brk` moved to the server on port +1 using the parsed flag | Same move, but using the effective address above; `--inspect-wait` is moved too (it used to be inherited on the parent's port, fail to bind and not wait); `--inspect-publish-uid` is kept for the server's inspector, including for `next dev --inspect` | | `isolatedMemory` / `enableSourceMaps` | Applied to `NODE_OPTIONS` only | Also drop an inherited CLI `--max-old-space-size` / `--no-enable-source-maps`, so the option wins | | Thread `execArgv` | Not handled | Threads reject V8/per-process flags in an explicit `execArgv`. If they reject only inherited flags (which already apply process-wide), the thread is created once more without them; a rejected requested flag fails the call | **Build and dev changes** - A static worker that exits (e.g. `process.exit(78)` while prerendering) ends `next build` with the worker's exit code and a one-line message, as before. - Prerender early exit throws a `NEXT_EXPORT_ERROR` back to the parent instead of calling `process.exit` inside the worker. Output from concurrently rendering pages is silenced after the first failure. The CLI prints the message as-is, and deferred Turbopack warnings still follow it. - Build checks that are started early and awaited later are marked as handled, so shutting the worker down after another failure can't raise an `unhandledRejection` that hides the real error. - `export/worker.ts` clears the `staticPageGenerationTimeout` timer once each page export settles, instead of leaving a ref'd 60 s timer per page. - `type-check.ts` drops the `Call retries were exceeded` special case. - The `'use cache'` probe pool spawns workers on demand (resolving its TODO), and the static-paths and dev-validation workers keep one call at a time (`concurrencyPerWorker: 1`). - Standalone traces (`collect-build-traces.ts`) and the Turbopack server NFT (`next_server_nft.rs`) include the new worker child entrypoints instead of the jest-worker ones, and prune them outside standalone builds. - The cache-components-errors test helpers stop at the `exiting the build` line, because jest-worker's `Next.js build worker exited` line no longer exists. Snapshots are unchanged. ### Verification Unit tests (`lib/worker/worker-pool.test.ts`, `lib/worker.test.ts`, `server/lib/utils.test.ts`) and the real-process tests `test/unit/worker-pool-boot-failure.test.ts` and `test/unit/worker-pool-errors.test.ts` pass on Node.js **20.9.0, 20.20.2, 22.23.5 and 24.22.3** (206 tests). **Edge cases verified** - Worker lifecycle (real child processes and real threads unless noted) - A module that exits while loading, with `maxRetries` 0 and 2: rejects after 3 and 9 spawns, `onRestart` called 0 and 2 times, no worker left running; with a healthy worker alongside, all calls resolve. - A worker that keeps timing out while booting (mocked): spawned 3 times, then the call rejects. - A module that throws at load time: every queued call rejects with the load error. - An uncaught async error in a thread retries like a process crash and doesn't reject unrelated queued calls; a late thread error after `shutdownNow()` doesn't write to the closed stderr. - Thrown string, thrown `undefined`, BigInt or circular result, an error with a function property: only that call fails, and calls running alongside on the same worker succeed. - Call arguments that fail to serialize (circular, BigInt, `DataCloneError`): only that call rejects, and the queue keeps draining. - A module that leaves a `setInterval` running shuts down in about 40 ms without a force-kill, with and without `NEXT_CPU_PROF`. - 256 KB of stderr written in `teardown()` arrives in full in 50/50 runs, for processes and threads. - A detached grandchild holding the stderr pipe for 8 s doesn't delay shutdown beyond the 5 s cutoff. - `next build` with `getStaticProps` calling `process.exit(78)`: exit code 78 and the one-line message, for processes and threads. - Node.js flags, on all four Node.js versions against the built `Worker` - Parent with CLI vs `NODE_OPTIONS` `--max-old-space-size`/`--conditions`, `--prof`, `--inspect-wait` (simulated on 20.9.0, which doesn't have it), `--inspect` via CLI and `NODE_OPTIONS`, and `--max-old-space-size --prof --inspect-wait` combined. Tested for plain, dev-style and static-style (`debuggerPortOffset`, `isolatedMemory`, `enableSourceMaps`) workers, as processes and threads. All 42 calls per version succeed; CLI flags win, `--prof` works, and static workers' debugger is on base + 1. - `--no-enable-source-maps` with `enableSourceMaps`; `--inspect-port <n>` as a separate value; `--inspect=…:9800 --inspect-port 9810` (worker on 9811); `--debug-port` with a separate loopback value after a wildcard `--inspect-port` (worker on loopback); a requested `--stack-size` on a thread (rejected at once). - Debugger relocation in `next dev` (real processes, all four Node.js versions) - `NODE_OPTIONS=--inspect-wait=127.0.0.1:9650`: the server waits on 9651, logs `- Debugger port: 9651`, and serves `/` after a debugger attaches. - `--inspect-wait` on a wildcard host followed by a loopback `--inspect-port`: the server listens on loopback, port +1. - `--no-inspect` / `--no-inspect-brk` / `--no-inspect-wait` negations: no server inspector when the parent's is disabled; a running inspector whose break/wait flag was negated is moved as plain `--inspect`. - Host-only `--inspect-port=localhost`, and the parent's inspector failing to start because its port is taken: the server stays on loopback. - `next dev --inspect` with `NODE_OPTIONS=--inspect-publish-uid=stderr`: the server's `/json/list` returns 404 (new test `test/development/dev-inspect-publish-uid`). - A parent inspector opened at runtime on `0.0.0.0` with `next dev --inspect 127.0.0.1:<port>`: the server listens on the given loopback address; without `--inspect` the server gets none. - e2e - `test-start-turbo` / `test-start-webpack`: `sync-io-blocks-root`, `cpu-profiling-build` (exactly 2 static-worker profiles), `app-dir/build-output` (now also asserts the exit code). - `test-start-turbo`: `cpu-profiling` (start and production), `worker-restart`, `cache-components-errors/{client,sync-dynamic,sync-io-node-crypto}`, `next-server-nft`, `build-output-prerender`, `build-tracing-message`, `edge-configurable-runtime`, `instant-validation/{sync-io-and-blocking,head-and-reporting.partial-prefetching}`, `instant-validation-level-manual-warning`. - `test-dev-turbo` / `test-dev-webpack`: `cpu-profiling-dev`, `dev-validation-worker-source-maps`, `devtool-copy-button` (also with port 9229 already taken), `cli` inspect tests. - `cargo check -p next-api`. <!-- NEXT_JS_LLM --> <!-- fleet 1cf3e53b-4113-4024-911b-b33a4e07eecb --> --------- Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Author
Parents
Loading