NOTES / EVALS AND OBSERVABILITY

Building eval sets

What an evaluation case contains, where cases come from, why you should start from error analysis, how to cover more than the easy cases, and how to keep a held-out set honest.

UPDATED 18 NOV 2026#evals#datasets#error-analysis

What a case contains

An evaluation dataset is a set of examples you run the agent on, the same way every time. One case holds an input, whatever context the agent needs, and a description of what a correct result looks like.

Diagram: a JSON evaluation case with input, patient ID, expected tool, expected facts and tags, annotated with who checks each field

Diagram: my own. The patient ID and values are invented.

The important design choice is expected_facts. It records what must be true of a good answer (value 5.7, unit percent, dated 1 July 2026) and not the sentence that says so. That lets a code check or a judge accept any correct wording, which is the whole problem described in Why evals come first.

Tags look minor but pay off later. They let you report scores by category, so a weak spot isn’t averaged away.

Where cases come from

Most useful sets mix several sources.

Diagram: five sources of examples flowing into one eval dataset that splits into a development set and a held-out set

Diagram: my own.

Source Good for Watch out for
Written from requirements Behaviour you know must work Reflects what you imagined, not what users do
Production traces Realistic phrasing and mess Privacy: strip or avoid sensitive data
Synthetic cases Filling gaps cheaply Treat as unverified until a person has reviewed them
Past failures Preventing regressions Skews towards problems you have already seen
Expert-labelled examples Ground truth for judgement calls Slow and costly; needs clear guidelines

Phoenix, for one, supports versioned datasets of examples for experiments and evaluation3. Whatever tool you use, record where each case came from. It helps when you later ask why the set over-represents one kind of question.

Start from error analysis

It is tempting to begin by writing a rubric. Hamel Husain argues against that: a common mistake is writing the rubric before looking at any data, and the right first step is error analysis, where you read and annotate real traces1. It is the most human-intensive part of evals, and also the part that tells you what to test.

Diagram: read real traces, note what went wrong, group notes into failure categories, turn each into a case or a check

Diagram: my own, following the workflow described in 1.

The practical version is simple. Read a batch of real traces. Write a short note on each thing that went wrong. Group the notes into recurring categories. Each category becomes cases in your set and, where possible, a check. Categories come out of the data, so they match your product rather than a generic list of qualities.

Cover more than the easy cases

A set made only of straightforward examples gives a flattering and misleading picture. For a lab-lookup skill, a better set includes:

Case What it tests
One result exists Basic retrieval
Several results, unordered Picking the latest
No results Saying so rather than inventing a value
Different date formats Date handling
A similar test name (HbA1c versus glucose) Choosing the right test
Tool returns an error Handling failure honestly
Patient ID missing Asking, not guessing

Anthropic notes that writing evals early forces you to define what success means2. These rows are that definition made concrete, including what “success” is when the data is missing or the tool fails.

Development and held-out sets

Split the set. Use the development part freely while you tune prompts, tools and code. Keep the held-out part for checking, and look at it rarely.

The reason is the same as in machine learning generally. If you tune against every example, each fix quietly bakes in knowledge of those exact cases. Scores rise, but you can no longer tell whether the system got better or only got used to your test. A held-out set that you have not tuned against is the only honest check that improvements generalise.

When a held-out case exposes a real failure, you can promote it into the development set and write a fresh held-out case to replace it.

Keeping the set healthy

  • Version it. You need to know which dataset produced which score.
  • Tag every case by category so results can be sliced.
  • Record provenance: authored, from production, synthetic, or labelled by whom.
  • Review synthetic cases before trusting them as references.
  • Retire stale cases when requirements change, and say why.
  • Handle sensitive data deliberately if cases come from real traces.

Start small and reviewed rather than large and unchecked. A few dozen well-understood cases that you have actually read beat thousands you haven’t. Grow the set as online monitoring surfaces new kinds of failure. The case fields and coverage table here are my own adaptation of examples from the course4.

Sources

  1. [1]Automating Error Analysis — Hamel Husain · hamel.dev · 2026PARAPHRASED
  2. [2]Demystifying evals for AI agents — Anthropic · Anthropic Engineering · 2026PARAPHRASED
  3. [3]Arize Phoenix documentation — Arize AI · Arize AIPARAPHRASED
  4. [4]Evaluating AI Agents (short course) — John Gilhuly and Aman Khan · DeepLearning.AI, with Arize AISTUDY SOURCE; EXAMPLES AND DIAGRAMS ARE MY OWN

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.