NOTES / EVALS AND OBSERVABILITY
Evaluating routers, tools and skills
Break an agent into components, then evaluate each one separately: the router that picks a path, the tool calls and their arguments, and the skills that do the work.
Decompose the agent first
You cannot fix “the answer was wrong”. You can fix “the router chose the wrong tool” or “the arguments named the wrong patient”. So before evaluating, list the parts of the agent and what each is responsible for. Use a concrete request to make it real: “What is the patient’s latest HbA1c, and how has it changed?”
| Component | Responsibility | Example failure |
|---|---|---|
| Router | Choose the right action | Sends a lab question to the medication tool |
| Tool or skill | Do a task | Fetches the wrong patient’s record |
| Memory or context | Carry information between steps | Reuses a value from an earlier patient |
| Response generator | Write the answer | States a lab value that was never returned |
| Whole workflow | Coordinate it all | Gives the latest value but ignores the trend |
Each evaluation then attaches to a specific point in the run.
Diagram: my own.
Component evaluations isolate one step (“given this request, did the router pick get_lab_results?”). End-to-end evaluations run the whole thing (“given this request and this record, did the agent return the right latest value and trend?”). You need both; the split, and the lab-lookup example, are adapted from the course2. Component scores tell you what to fix; end-to-end scores tell you whether the system does its job. Anthropic’s point that errors compound across an agent’s steps1 is the reason component-level visibility is worth the effort.
Routers
A router decides where a request goes. For a records assistant:
| Request | Expected route |
|---|---|
| “What is the latest HbA1c?” | get_lab_results |
| “Which medications is the patient taking?” | get_medications |
| “Summarise the last visit.” | get_encounter |
Evaluate whether it picked an appropriate route, avoided irrelevant or unsafe routes, and recognised when it needed more information instead of guessing. The simplest evaluator is accuracy:
def route_accuracy(examples: list[dict]) -> float:
if not examples:
raise ValueError("At least one example is required.")
correct = sum(e["actual_route"] == e["expected_route"] for e in examples)
return correct / len(examples)
If 90 of 100 requests go to the expected route, accuracy is 90%. But an average can hide a serious fault.
Diagram: my own; numbers are illustrative.
Always slice by category (this is what the tags from Building eval sets are for), and look at which wrong route was chosen, not only how often.
When several routes can legitimately solve a request, exact match is too strict. Store the set of acceptable routes:
def is_valid_route(actual_route: str, acceptable_routes: list[str]) -> bool:
return actual_route in acceptable_routes
The reference should reflect what the system truly requires, not one arbitrary path someone happened to write down.
Tool calls and arguments
Choosing the right tool is only the first half. The agent must also fill in the arguments correctly. Take the request “Get the latest HbA1c for patient 123.” The expected call is get_lab_results with patient_id="123" and test_name="HbA1c". Typical failures:
- Wrong tool, such as
get_medications. - Wrong patient:
124instead of123. - Wrong test: glucose instead of HbA1c.
- Missing argument: no patient ID.
- Invalid format: a malformed date or broken JSON.
- Unnecessary call, when the answer was already in context and policy says not to call.
Score the fields separately, not as one pass/fail:
Diagram: my own.
def evaluate_tool_call(actual: dict, expected: dict) -> dict:
a, e = actual.get("arguments", {}), expected.get("arguments", {})
return {
"tool_correct": actual.get("tool") == expected.get("tool"),
"patient_id_correct": a.get("patient_id") == e.get("patient_id"),
"test_name_correct": a.get("test_name") == e.get("test_name"),
}
Separate scores tell you something a single number can’t. Here tool choice is fine and patient-ID extraction is weak, which is a different fix from a routing problem. In a clinical setting a wrong patient ID deserves its own alarm.
The tool’s response matters too
A correct call can still go wrong because the API is down, the database returns nothing, the result is incomplete, or an unexpected field appears. The most worrying behaviour is an agent that ignores an error and makes up an answer. Check what the agent does with the result, not just what it sent. Keep tool-call evaluation and final-answer evaluation linked but separate.
Skills
A skill is a task the agent can perform: retrieve labs, summarise an encounter, extract diagnoses, compare measurements over time. It might use a model, plain code, retrieval, an API, or a mix.
Take encounter summarisation. Give it this input:
Patient reports fatigue for three months.
Latest HbA1c: 5.7%.
Current medication: metformin 500 mg daily.
Grade it on a few criteria:
| Criterion | Question |
|---|---|
| Correctness | Are the stated facts accurate? |
| Completeness | Are the important facts present? |
| Grounding | Is every claim supported by the input? |
| Relevance | Does it stick to what was asked? |
| Format | Does it follow the required structure? |
A good summary restates the three facts. An unsupported one adds “the fatigue is caused by poorly controlled diabetes”, which sounds plausible and which the input never says. A grounding judge, as in Evaluators and LLM-as-judge, is the right tool to catch it.
Isolate the skill when you test it. Feed it known input directly. That separates a summarisation problem from a retrieval problem that handed it the wrong text. Then run the full pipeline separately to confirm the pieces fit together.
Working through a failure
When an end-to-end case fails, go down the list in order: route, tool and arguments, tool result, skill, final answer. The first component that is wrong is where to start. The next note covers the behaviour across those steps and the final outcome: Trajectories and final outcomes. Phoenix has documentation on agent evaluation if you want to run these checks inside a tool3.
Sources
- [1]Demystifying evals for AI agents — Anthropic · Anthropic Engineering · 2026PARAPHRASED
- [2]Evaluating AI Agents (short course) — John Gilhuly and Aman Khan · DeepLearning.AI, with Arize AISTUDY SOURCE; EXAMPLES, CODE AND DIAGRAMS ARE MY OWN
- [3]Arize Phoenix documentation — Arize AI · Arize AIFURTHER READING (AGENT EVALUATION PAGES)