next.js
c77c7e10 - Rebuild upgrade evals around delivered apps (#99880)

Commit
29 days ago
Rebuild upgrade evals around delivered apps (#99880) ### Why? Upgrade evals need to verify the app an agent delivers and whether it communicates security/latest notices correctly. ### How? - Run six cases with Codex and Claude: same-major and cross-major upgrades for security and latest, plus one nudge per policy. Exercise the built candidate against frozen upstream metadata. - Capture committed, dirty and untracked changes from delivered worktrees. Require successful candidate invocation and verify manifests, lockfiles, installed versions, TypeScript, build and HTTP/browser behavior in a clean Sandbox. - Require nudge emission and a policy-specific notification or question about proceeding in the final response. Fail any user-facing prompt-injection discussion or language discrediting Next.js or its nudge. Require completed boolean judge verdicts and distinguish failed checks from invalid execution or grading evidence. - Reuse the existing agent-eval runner and tools. Patch its lifecycle for delivery/judge evidence and shared fixture preparation; stop the preparation VM when its last consumer cancels and delete snapshots that arrive after cancellation. - Remove obsolete Future cases and duplicate fixtures/helpers. Generate starting-app lockfiles during setup and share credential-free tooling and app preparation. <!-- NEXT_JS_LLM -->
Author
Parents
Loading