NOTES / EVALS AND OBSERVABILITY

Why evals come first

Why testing an LLM application is different from testing ordinary code, the three levels you can evaluate at, and the loop every later note plugs into.

UPDATED 18 NOV 2026#evals#agents#testing

Tests versus evals

An ordinary unit test pins down one right answer: assert add(2, 3) == 5. LLM applications break that habit in two ways. Several different answers can be correct, and a fluent answer can still be wrong.

Suppose a clinical-records assistant is asked for a patient’s latest HbA1c. It might say “The patient’s most recent HbA1c is 5.7%.” or “The latest recorded HbA1c was 5.7 percent.” Both carry the same fact. A string comparison would pass one and fail the other, so what you need is a check on the fact, not the wording.

Diagram: two differently worded answers; exact string match passes one and fails the other, while a fact check passes both

Diagram: my own. All patient data in these notes is invented.

Anthropic’s working definition is a good place to start: an eval is a test for an AI system, where you give it an input and then apply grading logic to the output to measure success1. The grading logic is the part that has to be rethought.

Three things you can evaluate

“Evaluation” covers three different questions, and mixing them up is a common source of confusion.

Diagram: three cards for model, application and component evals, with what each asks and an example

Diagram: my own, using the model-versus-application distinction described in 2.

Model evals measure what the base model can do in general. Public benchmarks such as MMLU and HELM are of this kind2. Application evals ask a different question: does our pipeline do its task, on our data, for our users2. A model can score well on a benchmark and still fail your product, because your product feeds it the wrong context, calls the wrong tool or combines results badly.

Component evals sit one level below. They test a single part, such as the retriever or the router, so that a failure points at something you can fix. The rest of this section is mostly about application and component evals.

Why agents are harder

A chat completion is one input and one output. An agent runs a chain of decisions1:

  1. Interpret the request.
  2. Choose a tool.
  3. Write the tool’s arguments.
  4. Read the result.
  5. Decide whether to act again.
  6. Write the final answer.

Anthropic points out that agents call tools over many turns and change state as they go, so mistakes can propagate and compound1. Two consequences follow:

  • A correct final answer does not prove every step was sound. The agent may have got there by luck or by wasteful detours.
  • A wrong final answer could have started at any step.

So agent evaluation looks at the outcome and at the behaviour that produced it. Evaluating routers, tools and skills and Trajectories and final outcomes take that apart.

The loop

Everything in this section hangs off one loop. You collect examples, run the agent on them, evaluate the results, inspect the failures, change the system, and run the same examples again.

Diagram: a loop of collect examples, run the agent, evaluate results, inspect failures, improve the system

Diagram: my own.

The point is to replace “the answer looks fine to me” with repeatable evidence about what works, what fails and whether a change helped. Anthropic notes that writing evals early forces a team to state what success means, and that teams without them tend to find problems only in production, where fixing one failure can create another1. The course this section draws on follows the same arc: trace the agent, evaluate it systematically, then improve it3.

How the section fits together

Step in the loop Note
See what the agent did Tracing and operational monitoring
Collect examples Building eval sets
Decide how to grade Evaluators and LLM-as-judge
Grade the parts of an agent Routers, tools and skills and trajectories and outcomes
Compare changes Experiments and eval-driven development
Keep learning from real use Online evals and A/B tests

Retrieval quality has its own metrics and lives in RAG and retrieval.

Sources

  1. [1]Demystifying evals for AI agents — Anthropic · Anthropic Engineering · 2026PARAPHRASED
  2. [2]The mirage of generic AI metrics — Hamel Husain · Decoding ML (guest post)PARAPHRASED
  3. [3]Evaluating AI Agents (short course) — John Gilhuly and Aman Khan · DeepLearning.AI, with Arize AISTUDY SOURCE; EXAMPLES AND DIAGRAMS ARE MY OWN

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.