Keep requests through middleware in a single OpenTelemetry trace (#100136)
### What?
Requests that pass through middleware / `proxy.ts` now produce **one
OpenTelemetry trace**, instead of two or three.
Before, with OTEL configured in `instrumentation.ts`:
| Request | No outer span | With an outer span (APM, custom server) |
|---|---|---|
| Middleware continues to render (`next()` / rewrite) | 3 traces (2
before 16.3) | 2 traces (1 before 16.3) |
| Middleware redirect / direct response | 2 traces (1 before 16.3) | 2
traces (1 before 16.3) |
After: 1 trace in every case.
### Why?
Two independent causes:
1. **Long-standing (13.5 → canary), #91282.** When middleware lets a
request continue, the router server runs `BaseServer.handleRequest`
twice: a middleware-invoke pass, then a render pass. Each pass extracts
its context from the incoming request headers. Without an incoming
`traceparent`, each pass started its own root trace.
2. **16.3 regression from #95306.** In-process Node.js middleware ran
under `ROOT_CONTEXT` to make `Middleware.execute` a sibling of
`handleRequest`. That also detached it from any outer span, so the
middleware span always started an extra trace.
Customers upgrading 16.2 → 16.3 see this as "doubled traces".
### How?
When middleware lets a request continue, the render is always **nested
inside the middleware span**, in every host:
```
next start / next dev:
handleRequest GET (bubble, root)
└─ middleware GET
└─ GET /page
separately-invoked middleware (Vercel, adapters, direct entrypoints), no parent:
middleware GET (root)
└─ GET /page
host span / incoming traceparent P:
P
└─ middleware GET (next start: P → handleRequest GET → middleware GET)
└─ GET /page
```
**Middleware → render.** The middleware adapter forwards the
`Middleware.execute` span's own W3C context (`traceparent` /
`tracestate`, via the configured propagator) as request-header overrides
on `next()` and rewrite responses. Every host already applies these
overrides to the downstream request, including the `next start` router
server, so no new host protocol is needed.
- Override lists replace the whole downstream header set. So:
- **No user overrides:** the adapter lists every request header the
middleware received, plus the trace headers.
- **User overrides present:** they are kept, and the trace headers are
added.
- **User-set trace headers:** a `traceparent` / `tracestate` the
middleware explicitly set (different from the one it received) wins. A
value copied unchanged from the request, as with `new
Headers(request.headers)`, is replaced so the render still nests.
- Redirects and direct responses are untouched.
**Request → middleware (self-hosted).** Once the middleware-invoke
pass's `handleRequest` span is active, its W3C context is written onto
the request headers. The middleware adapter (Node.js and the Edge
sandbox) extracts it, so the middleware span is a child of that pass.
The middleware-pass span keeps its existing name (`GET`); the route
stays on the render-pass span. If there is no middleware span to forward
(for example OTEL is registered only for Node.js while middleware runs
on Edge), the render falls back to this context and still stays in one
trace.
**#95306 regression.** The `ROOT_CONTEXT` detachment is removed, so an
outer span, an incoming `traceparent`, or the middleware-pass span now
parents Node.js middleware normally.
All of this only runs when OpenTelemetry is active and a valid span
context exists. Without OTEL, no headers are added.
One-time initialization traces stay separate by design: instrumentation
registration, route-module prewarming, and the once-per-process lazy
`RouteModule.prepare` of direct entrypoints.
### Tests
- **New deploy-capable suite
`test/e2e/app-dir/otel-middleware-single-trace`.** It runs Node.js
`proxy.ts` and Edge `middleware.ts` with `@vercel/otel`. Middleware
reports its active trace and span IDs; the render reports its active
trace ID and the `traceparent` it received. The test asserts one trace
and that the render's parent is the middleware span, so no collector is
needed. It runs in dev, start and deploy mode, with no deploy gate. It
covers:
- continue to a page and to a route handler, plus rewrite, redirect and
a direct response;
- a request the middleware doesn't match, and an incoming `traceparent`;
- user header overrides, including a deletion (the render still nests),
and a user-set `traceparent`;
- no internal `x-middleware-*` headers reaching the client.
- **Existing collector suite `test/e2e/opentelemetry/instrumentation`.**
The middleware tests now assert **real** trace IDs and the exact shape:
the render span's parent is the middleware span, the middleware span's
parent is the middleware-pass `handleRequest` (self-hosted) or the
external / host parent. They also check every span exported for the
request.
- **Setups:** default and direct-entrypoint hosting; Edge and Node.js
middleware; root context and incoming `traceparent`.
- **Requests:** page, route handler, rewrite and redirect.
- **New variant:** direct entrypoints with a host parent span.
- **Unit tests** for the override helper
(`forward-trace-context.test.ts`), including unchanged copied trace
headers.
### Verification
- The new suite's nesting checks and the collector shape checks fail
before this change (render parented to `handleRequest` or the external
span) and pass after it.
- New suite passes in dev (Turbopack), start (Turbopack, webpack), and
the same with `__NEXT_CACHE_COMPONENTS=true`.
- Full `test/e2e/opentelemetry/instrumentation/` passes in start mode
(Turbopack); the middleware tests also pass in dev (Turbopack) and start
(webpack).
- Passed on the previous revision (not re-run for the nesting change):
`test/e2e/opentelemetry/client-trace-metadata`,
`test/e2e/app-dir/otel-parent-span-propagation`,
`test/e2e/middleware-request-header-overrides`, and the W3C trace-header
fetch-cache test in `test/e2e/app-dir/app-static`.
- Deploy: an earlier revision's trace-ID assertions passed on Vercel
(default and Cache Components). The new parent-span assertion runs on
Vercel for the first time in this revision's CI.
Fixes #91282
<!-- NEXT_JS_LLM -->
<!-- fleet ce0ecb23-b14c-4679-92a9-e676130f618e -->
---------
Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
next.js
2e559938 - Keep requests through middleware in a single OpenTelemetry trace (#100136)
