NOTES / EVALS AND OBSERVABILITY

Tracing and operational monitoring

How to record what an agent actually did, using traces and spans, how to instrument it, how to read a trace when an answer is wrong, and which operational signals to watch.

UPDATED 18 NOV 2026#observability#tracing#opentelemetry#phoenix

Why the final answer is not enough

If all you log is the last message, a wrong answer is a dead end. You can see that it is wrong but not where it went wrong. Observability means being able to inspect what happened inside the system, and for an agent that means its decisions and intermediate actions, not only its output.

Take a clinical-records assistant that says the latest HbA1c is 6.4% when the newest result is 5.7%. That could be a retrieval problem, a selection problem or a generation problem, and each has a different fix. Only the trace can tell them apart.

Traces and spans

OpenTelemetry, the open standard most tracing tools build on, defines a span as a unit of work or operation, and a trace as the spans that belong to one request1. In the specification’s terms, a trace is a directed acyclic graph of spans whose edges are parent-and-child relationships2. For an agent, one user request becomes one trace, and each model call, tool call and processing step becomes a span.

Diagram: a trace drawn as a tree of spans against a time axis

Diagram: my own; timings are illustrative. Child spans sit inside their parents.

Nesting matters. The database query happens inside the lab lookup, which happens inside the request. That structure is what lets you see where time was spent and which step called which.

What a span records

Every span carries a context that includes the trace ID and its own span ID, which is how spans are stitched into one trace1. Spans also carry attributes, which are key-value pairs describing the operation1. AI tooling adds fields that matter for agents.

Diagram: the fields on every OpenTelemetry span, plus typical extra fields for AI spans

Diagram: my own. The left column is the OpenTelemetry basics1; the right column lists what AI-oriented instrumentation typically adds.

For a tool-using agent, the useful extras are the tool name and arguments, the model and its settings, the input and output, token counts and any error.

Be deliberate about what you record. In a clinical system, prompts, tool arguments and tool outputs can all contain sensitive data. Decide up front which fields are stored, which are redacted, who can see them and how long they are kept. This is a design decision, not something to bolt on after the traces already exist.

Instrumenting an agent

Instrumentation is the code that turns your agent’s operations into spans. The flow looks like this:

Diagram: agent runs, instrumentation library, spans exported, Phoenix, selecting examples, evaluation dataset

Diagram: my own.

Arize Phoenix is an open-source tool for experimentation, evaluation and troubleshooting of LLM applications, and it works with OpenTelemetry and OpenInference instrumentation3. As I understand it, OpenInference supplies the conventions for describing AI-specific operations (LLM calls, tool calls, retrieval) on top of OpenTelemetry4. Phoenix is one choice for the viewer; the same spans can go to any OpenTelemetry-compatible backend.

Libraries can instrument many frameworks automatically. When you need a span of your own, the OpenTelemetry API is small:

from opentelemetry import trace

tracer = trace.get_tracer("preconsult-agent")

def get_lab_results(patient_id: str, test_name: str):
    with tracer.start_as_current_span("get_lab_results") as span:
        span.set_attribute("tool.name", "get_lab_results")
        span.set_attribute("tool.test_name", test_name)  # avoid raw identifiers
        rows = query_database(patient_id, test_name)
        span.set_attribute("tool.result_count", len(rows))
        return rows

Note that the example records the count of results rather than the rows, and does not store the patient ID. Attribute names above are mine; follow the conventions of whatever instrumentation library you adopt.

Reading a trace

When an answer looks wrong, walk the trace in the order the agent worked:

  1. The original request.
  2. The tool the agent chose.
  3. The arguments it passed.
  4. What the tool returned.
  5. Any later model calls.
  6. The final answer.
  7. Errors, latency and token use along the way.

Here is the 6.4% versus 5.7% case as a decision tree:

Diagram: decision tree asking whether the correct value was in the tool result and whether the agent selected it, ending in retrieval, selection or generation failure

Diagram: my own.

A concrete trace for this case might look like the following. The newest value is in the result, so the fault is not retrieval:

{
  "tool": "get_lab_results",
  "tool_arguments": { "patient_id": "p_001", "test_name": "HbA1c" },
  "tool_result": [
    { "date": "2026-03-01", "value": 6.4, "unit": "%" },
    { "date": "2026-07-01", "value": 5.7, "unit": "%" }
  ],
  "final_answer": "The latest HbA1c is 6.4%."
}

Both rows came back, and the agent picked the first. That points at selection, and a likely fix is to sort by parsed date in ordinary code before the model sees the rows. Patient ID and values are invented.

Operational monitoring

Traces also feed the numbers you watch to keep a system healthy. These are operational signals, meaning whether the system is running, fast and affordable:

Signal What it tells you Watch out for
Latency per span Which step is slow Averages hide slow tails
Tokens and cost per task Where spend goes One looping task can dominate
Exceptions and failed requests Reliability Silent failures that return an empty answer
Tool error rate, per tool Which integration is flaky A low overall rate can hide one broken tool
Repeated calls and retries Loops or confusion Same tool, same arguments, many times

Whether the answers are good is a separate question, covered in Online evals.

Tracing is evidence, not a score

A trace shows what happened. It does not say whether that was acceptable. Judging that needs an evaluator against a stated requirement, which is the subject of Evaluators and LLM-as-judge. The course this note draws on adds observability to an agent first and then evaluates each step5, which is the order used in this section: you have to be able to see the steps before you can grade them.

Sources

  1. [1]Traces (OpenTelemetry concepts) — OpenTelemetry Authors · OpenTelemetryPARAPHRASED
  2. [2]OpenTelemetry specification overview — OpenTelemetry Authors · OpenTelemetryPARAPHRASED
  3. [3]Arize Phoenix documentation — Arize AI · Arize AIPARAPHRASED
  4. [4]OpenInference — Arize AI · GitHubFURTHER READING
  5. [5]Evaluating AI Agents (short course) — John Gilhuly and Aman Khan · DeepLearning.AI, with Arize AISTUDY SOURCE; EXAMPLES AND DIAGRAMS ARE MY OWN

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.