fix: status code to avoid navigation with empty props (#60968)
### What?
Currently, when a middleware is active and the path results in a 404,
the server responds with a 200 status code to data requests. We propose
a change to ensure that a 404 status code is returned in these
situations, as expected, solving production issues for the navigation
router.
### Why?
The client data requests (identified with `_next/data` path &
`x-nextjs-data` header) get answered the status code 200. The code below
is executed when there is a version skew for the data requests (e.g
prefetch in production).
https://github.com/vercel/next.js/blob/4125069840ca98981f0e7796f55265af04f3e903/packages/next/src/server/lib/router-server.ts#L217-L227
The version skew consists in the client side version identified with the
buildId in `__NEXT_DATA__ ` tag to be obsolete compared to the server
version (`BUILD_ID` file).
In the case of prefetching, this leads to the code above being executed,
therefore the `prefetch-reducer.ts` handles the response as valid and
sets it in its cache. Which ultimately triggers a navigation with empty
prop, resulting in erroneous behaviours reported in issues and in our
production websites:
https://github.com/vercel/next.js/blob/4125069840ca98981f0e7796f55265af04f3e903/packages/next/src/client/components/router-reducer/reducers/prefetch-reducer.ts#L54-L74
By switching the response to a 404, we trigger this code in the fetch
(`fetchServerResponse`). This change prompts a hard navigation
(mpaNavigation), effectively refreshing the client version and
resynching it with the server version.
https://github.com/vercel/next.js/blob/4125069840ca98981f0e7796f55265af04f3e903/packages/next/src/client/components/router-reducer/fetch-server-response.ts#L125-L134
### How?
We simply update the status code to 404 here:
https://github.com/vercel/next.js/blob/4125069840ca98981f0e7796f55265af04f3e903/packages/next/src/server/lib/router-server.ts#L223
Closes: https://github.com/vercel/next.js/pull/60785
Closes: https://github.com/vercel/next.js/issues/59295
Fixes: https://github.com/vercel/next.js/issues/47516
---------
Co-authored-by: JJ Kasper <jj@jjsweb.site>
