eval: split Next.js evals by purpose (#100742)
## Summary
Split the public Next.js evals by purpose using unpublished pass@1
measurements across Next.js 16.4.1, 16.4.2, and 16.5.0-canary.17.
`regression` preserves behavior already saturated across frontier and
weaker models, `competency` contains verified tasks that still
distinguish model capability, and `audit` is the starting point for new
or unverified tasks and tasks whose grader, documentation sensitivity,
or stability needs review. In this initial split, the five competency
tasks are saturated for frontier models but still distinguish weaker
models.
Percentages below are pass@1 with `AGENTS.md`, shown as frontier /
weaker.
### Regression
- `000-app-router-migration-simple` — 96% / 89%; saturated across tiers.
- `021-avoid-fetch-in-effect` — 96% / 100%; saturated across tiers.
- `022-prefer-server-actions` — 100% / 100%; fully saturated.
- `023-avoid-getserversideprops` — 100% / 89%; saturated across tiers.
- `024-avoid-redundant-usestate` — 100% / 100%; fully saturated.
- `025-prefer-next-link` — 100% / 100%; fully saturated.
- `027-prefer-next-image` — 100% / 100%; fully saturated.
- `031-proxy-middleware` — 96% / 89%; saturated with docs and useful as
a docs regression guard.
- `033-forbidden-auth` — 100% / 100%; fully saturated.
- `034-async-cookies` — 100% / 100%; fully saturated.
- `036-after-response` — 100% / 100%; fully saturated.
- `037-updatetag-cache` — 100% / 100%; fully saturated.
- `039-indirect-proxy` — 100% / 100%; saturated with docs and useful as
a docs regression guard.
- `040-instant` — 96% / 100%; saturated across tiers.
- `042-enable-ppr` — 100% / 100%; fully saturated.
- `043-view-transitions` — 100% / 100%; fully saturated.
- `045-build-a-nextjs-app` — 91% / 89%; failures did not reproduce
consistently.
- `048-prefetch-url-data` — 100% / 100%; saturated with docs and useful
as a docs regression guard.
- `052-cache-components-session-isolation` — 100% / 100%; fully
saturated.
### Competency
- `028-prefer-next-font` — 100% / 78%; saturated for frontier models
only.
- `032-use-cache-directive` — 100% / 67%; weaker models have
reproducible failures.
- `049-prefetch-stage` — 96% / 67%; weaker models still show a real docs
and capability gap.
- `053-cache-components-url-shell` — 92% / 75%; saturated for frontier
models only.
- `054-cache-components-empty-static-params` — 100% / 78%; saturated for
frontier models only.
### Audit
- `026-no-serial-await` — 88% / 67%; `AGENTS.md` reduced the pass rate.
- `029-use-cache-directive` — 87% / 89%; weaker models moved from 0%
without docs to 89% with them.
- `030-app-router-migration-hard` — 83% / 67%; docs hurt frontier models
but helped weaker models.
- `035-connection-dynamic` — 78% / 100%; frontier failures were flaky
rather than reproducible.
- `038-refresh-settings` — 83% / 33%; genuinely hard, but its grader
needs review.
- `041-optimize-ppr-shell` — 83% / 75%; most failures were flaky or
unconfirmed.
- `051-prefetch-until-navigation` — 87% / 67%; genuine failures remain,
but the grader needs tightening.
`044-uses-nextjs` is marked `publish: false` because it measures
framework preference and belongs in the a0 AEO suite.
The runner adds `pnpm eval --suite <name>` while preserving individual
eval and `--all` behavior. Suite selection remains independent from the
baseline and `AGENTS.md` treatments. This PR does not change task
prompts, graders, model configuration, or existing results.
## Verification
- `pnpm --filter=next build`
- `pnpm typescript`
- `pnpm eval --suite regression --dry`
- `pnpm eval --suite competency --dry`
- `pnpm eval --suite audit --dry`
- `pnpm eval agent-044-uses-nextjs --dry`
- Unable to complete `pnpm build-all` because the current Cargo
workspace rejects `turbo-persistence` overriding `default-features`; the
focused Next.js package build passed.
<!-- NEXT_JS_LLM -->
