Prompt Regression CheckGuide

How to export your OpenAI Evals before they go read-only

Updated 2026-10-10 · by Hieu Tran, written with AI agents and checked against the sources below

OpenAI is deprecating its hosted Evals: existing evals become read-only on 31 October 2026 and the platform shuts down on 30 November 2026. OpenAI's docs don't offer an export button, so if your test cases and past results live there, copy them out now.

What there is to export

• The eval: its data source config and its testing criteria (the graders: string checks, text similarity, model graders, Python graders).
• Runs: every run with the model, prompt template and result counts.
• Output items: for each test row, the input, the model's output and each grader's result. This is the part you can't rebuild later, because it records how past models behaved.
• Source files you uploaded for runs. Some may not be downloadable, but every row a run used is also inside its output items.

Export by API (manual)

All of it is readable through the Evals API with your API key. Each list endpoint is paginated: pass limit=100 and after=<last id> until has_more is false.

1. List evals: GET /v1/evals
2. For each eval, list runs: GET /v1/evals/{eval_id}/runs
3. For each run, list output items: GET /v1/evals/{eval_id}/runs/{run_id}/output_items
4. Optionally download a run's source file: GET /v1/files/{file_id}/content
5. Save everything as JSON, and keep the unique datasource_item rows as a JSONL dataset you can reuse.

Where to run them next

• promptfoo (open source; OpenAI's docs point to it): map string checks to equals/contains assertions and model graders to llm-rubric. Python graders need porting by hand.
• Langfuse and other eval platforms publish their own migration guides from OpenAI Evals.
• Whatever you choose, keep the raw export: it's your record of how each model version scored.

Don't wait for the last week

After 31 October you can still read evals but not create runs, and from 30 November the API is gone. Exports of large evals take a while because output items are paged 100 at a time, so run the export well before the shutdown.