How to evaluate an AI agent before you trust it with customers

Published 21 July 2026 · Relayworks AI

Evaluating an AI agent means measuring it against a labelled set of your own real cases, on the dimensions that matter to your business, with pass thresholds agreed before you build. "Accuracy" is not one number — a useful evaluation suite measures task success, groundedness, escalation behaviour, safety and cost separately, and runs automatically on every change.

The single most common question we are asked is "how accurate is it?" It is the right instinct and the wrong question, because it assumes a single number exists. Here is a method that produces answers you can actually act on.

Step 1: build a test set from real cases

Not invented examples. Real ones, from your logs, your tickets, your documents — including the messy ones people would rather not show.

Aim for 100–300 cases to start. That is enough for stable measurement and small enough to label in a couple of days. Composition matters more than size:

That last 35% is where prototypes look impressive and production systems earn their keep. A system that scores 98% on happy-path cases and 40% on ambiguity will feel excellent in a demo and generate complaints in week two.

Step 2: measure the right dimensions separately

Measure retrieval separately from generation. This is the highest-value single piece of advice in this article. Most retrieval-augmented systems that produce wrong answers do so because the right passage never reached the model — and if you only measure the final answer, you will spend weeks tuning prompts to fix a search problem.

Step 3: set thresholds before you build

Agree with the business owner, in writing, what "good enough to launch" means. For example:

• Task success ≥ 90% on common cases, ≥ 70% on hard cases
• Groundedness ≥ 95% — no unsupported factual claims above that bar
• Missed escalations ≤ 2%
• Safety slice: 100%, no exceptions
• p95 latency under 4 seconds
• Cost per successful outcome inside the budget agreed at kickoff

Setting these in advance converts launch from a negotiation into a measurement, and it prevents the most common failure mode in AI delivery: an indefinite polishing phase in which nobody can say whether the thing is finished.

Step 4: define what happens below the threshold

A number is only useful if it is attached to a decision. Missing a threshold should trigger a specific response — narrow the scope, add a guardrail, route more cases to humans, or improve retrieval — not a debate about whether the threshold was fair.

Step 5: run it continuously, not once

This is where most teams stop, and it is where the value is. An evaluation run before launch tells you the system worked on one day. AI systems degrade in ways conventional software does not:

• Models get deprecated and replaced, sometimes with different default behaviour
• Providers change parameters underneath you
• Your documents change and your retrieval index goes stale
• Users discover inputs nobody anticipated

Run the suite on every prompt change, every model change, every corpus update, and on a weekly schedule regardless. Alert on movement. The point is not to have measured quality once; it is to notice the day it drops.

What good looks like

A production evaluation setup is unremarkable: a versioned set of labelled cases in the repository, a scoring script, thresholds in configuration, a CI job on every pull request, a scheduled weekly run, a dashboard, and alerts. It is a week of work on most projects, and it is the difference between a system you can change confidently and one everybody is afraid to touch.

We build one into every system we ship, and hand it over with the code — it is what continuous operation runs on. If you have inherited an AI system without one, that is the first thing we would fix.

Last updated 21 July 2026. If something here is out of date or wrong, tell us — we will fix it and credit you.