NOTES / EVALS AND OBSERVABILITY
Online evals and A/B tests
Evaluating an agent on live traffic: what to score, how to sample, how production failures flow back into your eval set, and where A/B tests fit.
Offline and online
Everything so far has been offline: a fixed dataset, run before release. Offline evals are controlled and repeatable, which is why they’re the right place to compare versions. They can’t tell you what real users will do. Online evals score the agent on live traffic.
Diagram: my own.
Anthropic’s guide describes the pattern of catching problems only in production as a reactive loop in which fixing one failure creates others1. Online evaluation doesn’t replace offline evals. It makes sure that what you learn from production lands in the offline set.
What to score
Tracing and operational monitoring covers whether the system is running: latency, cost, exceptions, tool error rates. Online evals cover whether the answers are good:
| Signal | Example |
|---|---|
| Grounding | A judge checks sampled answers against the records the agent actually retrieved |
| Tool use | Wrong arguments, ignored errors, repeated calls |
| Task completion | Did the user get what they came for |
| User feedback | Explicit ratings, rephrasing, abandonment |
| Unfamiliar inputs | Requests unlike anything in the eval set |
Live traffic usually has no reference answer, so these evaluators tend to be reference-free: they compare the answer to the evidence the agent saw, or to a rule, rather than to a known-correct value. That makes them blunter than offline checks, and an agent can be grounded in stale data and still wrong, as in Trajectories and final outcomes.
Sampling and cost
You rarely need to score every trace. Score a sample, and make it a sensible one: include a random slice for an unbiased view, and over-sample the interesting cases (errors, long runs, negative feedback, unusual inputs). Judges cost money and add latency, so run them asynchronously rather than in the user’s path.
Privacy deserves care. Sending real clinical text to a third-party judge model is a data-handling decision. Check your obligations, redact what you can, and prefer a judge that runs inside the same boundary as the data.
The production-to-evaluation loop
The valuable part is the loop.
Diagram: my own.
Example: users keep asking, “Compare this month’s results with the previous two reports.” Your offline set only tests a single report. Sampled traces show the agent struggling with multi-report requests. You review a handful, label them, add representative cases, fix the behaviour, test it offline against the new cases, and ship. The next time a similar request fails, you’ll know.
Human review is central. Husain’s advice is to get the domain expert doing error analysis as soon as possible, with developers helping to spot broken tool calls and missing context2. Automated scores decide what to look at, while people decide what the failures mean.
A/B tests
Once a change passes offline evals, you can expose it to a fraction of real traffic and compare outcomes with the current version. Choose the outcome metric before you start, split users rather than individual messages so the experience is consistent, and watch safety-relevant signals as well as the headline number. An A/B test is a final check against real behaviour, not a substitute for the offline suite: it is slower, riskier for the users in the test group, and noisier.
A first set of measurements
For a pre-consult style agent, a first evaluation pass could cover:
- Router: did it choose the right workflow?
- Retrieval: did it fetch the right patient’s data?
- Tool arguments: were ID, test and date filters correct?
- Trajectory: were the necessary steps completed?
- Final answer: were key facts correct, complete and grounded?
- Operations: latency, errors and cost.
Start with a small, reviewed dataset, run it offline, and expand it from what production teaches you. The aim is a feedback loop that makes every change measurable4. Phoenix documents online evaluation if you want to see a worked setup3.
Sources
- [1]Demystifying evals for AI agents — Anthropic · Anthropic Engineering · 2026PARAPHRASED
- [2]Hamel Husain explains why AI evals fail before the evaluation begins — Sara Verdi · Arize AI · 2026PARAPHRASED
- [3]Arize Phoenix documentation — Arize AI · Arize AIFURTHER READING (ONLINE EVALS)
- [4]Evaluating AI Agents (short course) — John Gilhuly and Aman Khan · DeepLearning.AI, with Arize AISTUDY SOURCE; EXAMPLES AND DIAGRAMS ARE MY OWN