A regression set built from your real traffic runs on every pull request and every model upgrade. You see which answers changed, which got worse, and what it costs, in the PR. OpenAI's hosted Evals goes read-only on 31 Oct 2026; bring your evals with you.
We'll email you about early access, at most twice. No spam; reply STOP to opt out.
Free now: export your OpenAI Evals before they go read-only on 31 Oct (runs in your browser)
Keep the prompts that broke before as tests, with expected behaviour, format and cost limits.
A GitHub Action comments on the pull request: changed, better, worse, and cost per call.
Run the same set against a new model or provider version and get a diff before you switch.
On Hacker News, engineers call undocumented model regressions at the application layer the most frustrating kind.
Another: when you swap models or tweak prompts, regressions slip in: cost spikes, format drift, leaked PII.
Had this happen recently? Tell us what happened (4 questions, about 10 minutes, no sign-up).
Flat team pricing. The GitHub Action and an OpenAI Evals export stay free.
We'll email you about early access, at most twice. No spam; reply STOP to opt out.