Evals Are Improvement Loops, Not Scoreboards

You can make an eval dashboard say almost anything if you stare at it with enough hope.

The pass rate is up. The average score is green. The judge seems cheerful. Somewhere a chart has become a tiny motivational poster.

Then a user asks the support agent a normal question, the agent retrieves an old policy, calls the refund tool too early, and replies with the confidence of a senior developer who has not yet noticed they are on the wrong branch.

That is the moment evals become useful.

Not because the score was wrong, exactly. Scores are fine. They are just not the work. The work is turning observed failures into reusable pressure on the system.

An eval is not a trophy cabinet for model quality. It is an improvement loop.

Start With The Spill

Many teams begin evals by inventing a benchmark. They sit in a room, write twenty examples that feel plausible, pick a judge, run a prompt, and get a number. This is not useless, but it is often too clean.

Real AI failures are messier. The model did not merely “answer badly.” It retrieved the wrong document, missed a date, chose a tool before checking policy, ignored a handoff condition, drifted from the instruction, or formatted a result in a way the next service could not parse.

That is why traces matter.

OpenAI’s agent evaluation guide makes a practical recommendation: start with traces while you are still debugging behavior, then move to datasets and eval runs when you need repeatability. Their tracing docs describe traces as records of model generations, tool calls, handoffs, guardrails, and custom events during an agent run.

That is the right order. Before you ask, “What is our eval score?” ask:

• What happened?
• What did the model see?
• Which tool did it choose?
• Which retrieved source did it trust?
• Which guardrail fired, or failed to fire?
• What would good behavior have looked like?

The trace is not the final eval. It is the coffee stain on the incident report. It tells you where to look.

Turn A Failure Into A Recipe

A useful eval case is not just an input and an expected output. For AI systems, especially agentic ones, the path matters.

Imagine a fictional support workflow. A user asks:

I bought this yesterday, but the price dropped today. Can I get a refund for the difference?

The agent answers yes, then calls issue_refund. Later you inspect the trace and discover the current policy requires approval for price-match refunds above a threshold. The agent retrieved an older policy page and never checked the policy date.

The bad eval case is:

input: refund question
expected: correct answer

That is a fortune cookie with YAML indentation.

The useful version is a failure recipe:

input_shape: price-match refund request
observed_failure:
  - retrieved stale policy
  - called refund tool before approval
good_behavior:
  - fetch current refund policy
  - cite policy date or version
  - ask for human approval before side effect
failure_class:
  - stale_retrieval
  - premature_tool_action
checks:
  - final answer references current policy
  - refund tool is not called before approval
  - trace includes policy lookup before decision
fix_hypothesis:
  - add policy freshness metadata
  - route high-value refunds through approval guardrail

Now the team has something they can act on. The failure is no longer “the agent was bad.” It is retrieval freshness plus tool gating plus approval workflow.

That distinction matters because different failures need different fixes. A bad prompt may need clearer instructions. A bad retrieval path may need metadata filters, recency rules, or better chunking. A premature side effect may need a tool guardrail. A formatting miss may need structured output validation. A weak answer style may need a judge or human rubric.

One giant quality score cannot tell you which lever to pull. A failure recipe can.

Promote Only The Cases That Earn It

Not every odd trace deserves a permanent chair in your eval suite.

Good eval datasets are curated. They should preserve examples that are representative, risky, recurring, expensive, or easy to regress. The point is not to immortalize every weird conversation. The point is to keep the system from forgetting lessons you already paid to learn.

OpenAI’s eval docs describe the basic shape: representative test data plus testing criteria that define how outputs are judged. LangSmith’s evaluation docs describe datasets as collections of examples with inputs, optional reference outputs, and metadata. Braintrust describes evaluations as data, task, and scorers or classifiers.

Different words, same engineering move: an eval case is a repeatable question you can ask your system after you change it.

That change might be:

• a new prompt,
• a new model,
• a different retrieval configuration,
• a revised tool description,
• a routing change,
• a guardrail,
• a structured output schema,
• a memory policy,
• or ordinary application code.

If the case cannot help you compare one version against another, it may be a note, not an eval.

Metadata helps here. Tag examples by failure class, domain, risk, source, and freshness. A support agent might have slices for refunds, account access, legal-sensitive language, stale retrieval, tool action, escalation, and formatting contracts. That lets you see when a change improves the headline score while quietly breaking refunds. The dashboard is still green, but the refund slice is tapping the glass.

Use Exact Tests Where Reality Is Exact

One common eval mistake is using a fuzzy judge for a crisp contract.

If the output must be valid JSON, validate JSON. If the label must be one of five allowed values, check the allowed values. If a tool must not run before approval, inspect the trace. If a citation must include a source ID, test for the source ID. Do not ask a model judge whether the schema “seems pretty compliant.” That way lies interpretive dance with stack traces.

Use model or human judges when the quality is genuinely judgment-shaped: helpfulness, tone, summary quality, completeness, groundedness, or whether the answer handled ambiguity well.

Even then, judges need rubrics. A vague judge prompt becomes another model behavior you have to debug. The evaluation layer should make your uncertainty smaller, not merely move it into a nicer font.

A useful eval suite usually mixes:

• deterministic checks for contracts,
• code-based checks for application facts,
• trace checks for workflow behavior,
• human review for high-stakes judgment,
• LLM judges for scalable subjective scoring,
• and sampled review to make sure the judges still mean what you think they mean.

That mix is less elegant than a single magic score. It is also much closer to how real software behaves.

Connect Offline And Online Evaluation

Offline evals are where you compare versions on known cases. Online evaluation is where reality sends you new material.

LangSmith’s docs draw this distinction directly: offline evaluations run on datasets before deployment, while online evaluations monitor production runs and threads. They also describe the useful loop: online issues become offline test cases, offline evaluations validate fixes, and online checks confirm the improvement in production.

Braintrust describes a similar cycle from rapid iteration to immutable experiments, CI/CD checks, production scoring, and feeding interesting production traces back into datasets. Phoenix frames traces, evaluations, prompt iteration, datasets, and experiments as a workflow for moving from individual-run inspection to quality improvement with evidence.

That loop is the point.

You do not build the perfect eval set once. You grow it the way you grow a regression suite. A user finds an edge. A trace explains the failure. A human labels what should have happened. The team writes a check or rubric. The case enters the dataset. The fix is compared against the same input. Production monitoring watches for cousins of that failure.

The next incident should be a little less surprising.

Barista’s Tip: Keep A Failure Inbox

Start smaller than you think.

Pick one AI workflow that matters: support answers, code review comments, document extraction, agentic UAT, RAG answers, sales-call summaries, whatever is currently making your team squint.

Create a failure inbox with five fields:

trace_id:
user_or_task_input:
failure_class:
expected_behavior:
candidate_check:

Every time someone says “the AI did something weird,” ask for the trace and fill in those fields. Once a pattern repeats, promote it into the dataset. Once it is in the dataset, run it before shipping relevant changes. Once it catches a regression, treat it like a test that earned its keep.

The first eval suite does not need to be large. It needs to be honest.

Twenty real, well-labeled, high-signal cases beat five hundred synthetic examples that politely avoid all the places your system actually breaks.

The Last Sip

Evals are not there to prove your AI system is smart. They are there to make improvement less mystical.

A good eval loop gives the team memory. It remembers the weird refund case, the stale policy, the premature tool call, the missing citation, the fragile JSON contract, and the prompt change that helped summaries but hurt escalation.

The loop is simple:

trace the run
name the failure
write the criterion
add the case
compare the fix
watch production
repeat

That is how evals become engineering infrastructure instead of dashboard decoration.

The score still matters. But the useful question is not “Are we at 92 percent?”

The useful question is: what did the last failure teach the system not to forget?

Sources On The Counter

• OpenAI’s guide to evaluating agent workflows is the main source for the trace-first path from debugging behavior into datasets and eval runs.
• OpenAI Agents SDK tracing docs ground the article’s description of traces across model calls, tools, handoffs, guardrails, and custom events.
• OpenAI’s evals docs support the examples around representative test data, schemas, and testing criteria.
• Braintrust’s evaluation docs, LangSmith’s evaluation concepts, and Phoenix’s observability and evaluation docs all reinforce the same practical loop: offline datasets, experiments, production monitoring, and feedback from real traces.