Is your AI system working? How to evaluate it properly
The way to tell whether an AI system works is to measure it against a fixed test set built from real questions. For each question, what counts as correct is written down in advance. Results are scored by automatic checks and by someone who knows the work, and once the system is live, samples are reviewed on a regular schedule. Any change made without measuring can break the system as easily as improve it.
Below we set out, step by step, how an evaluation can be built. The system might be a chat assistant, a document summariser or an agent that takes actions on its own — the method is largely the same.
What is AI evaluation?
AI evaluation means measuring, in a repeatable way, how far a system’s outputs meet criteria defined beforehand. Repeatable is the important word. A measurement taken today should be possible to run again tomorrow, on the same questions, so the two results can be compared.
This is not our idea alone. The US National Institute of Standards and Technology (NIST) published its AI Risk Management Framework on 26 January 2023, and it names measuring as one of four core functions: govern, map, measure and manage. On 26 July 2024 NIST added a separate profile for the risks specific to generative AI.
Why a demo is not enough
A demo shows the system on a good day. A few prepared questions are asked, and the answers impress. Real use is messier: questions are vague, spelling slips, and people ask about things nobody thought of.
Two traits of language models widen that gap:
- They may not give the same answer to the same question every time. One correct answer does not mean the answer will be correct every time.
- A small change can affect unexpected places. Rewording one sentence in the instructions can fix refund questions whilst breaking invoice questions. A version update from the model provider can have the same effect.
So “we tried it and it works” is not a measurement. A measurement is running the same set of questions again after every change.
Which layers does an evaluation need?
No single method catches everything. Our recommendation is to build four layers together.
| Layer | What it catches | Who or what does it | When |
|---|---|---|---|
| Automatic checks | Format errors, missing fields, missing sources, banned phrases | Code | On every output |
| Test set | Accuracy on known questions, and regressions | Code plus expected answers | Before every change |
| Human review | Meaning, tone, domain knowledge, answers that are incomplete rather than wrong | Someone who knows the work | Before launch, then at regular intervals |
| Production monitoring | New kinds of question the test set lacks, drift | Logs, feedback, sample reviews | Continuously |
The first layer is the cheapest, and most teams skip it. If a system is asked for output with set fields — a ticket category and a priority, say — code can check that those fields hold valid values. If an answer must cite a source, code can check that the cited source is a document that actually exists.
How to build a test set
A test set is a representative sample of the questions the system will face. We suggest building one in these steps:
- Start with real questions. Collect genuine examples from the inbox, the support log and the sales team’s notes. Questions a team invents tend to be too tidy.
- Sort them into categories. Which topics do the questions fall into — refunds, invoices, delivery, product details? Make sure each category has enough examples.
- Add the hard cases. Ambiguous questions, ones that ask two things at once, ones written angrily, ones full of typos.
- Add questions with no answer. Questions the sources cannot answer. The correct behaviour here is for the system to say it does not know. We cover this in our piece on hallucination, privacy and human approval.
- Write the criterion for each question. Sometimes there is a single right answer. More often a criterion works better: “must include these three facts, must not include this phrase, must rest on this document”.
- Freeze and version the set. If the test set changes, results can no longer be compared with earlier ones. Adding questions is fine, but the record should say which version was used.
- Strip out personal data. Names, phone numbers and ID numbers in real records are masked before they enter the test set.
Starting small is fine. A short set that covers the categories well tells you more than a long one gathered at random. It grows over time as new kinds of question appear in production.
Which criteria should you use?
The criteria depend on what the system does. A single “accuracy score” is usually misleading.
| Type of system | What to measure |
|---|---|
| Assistant answering from documents | Correctness, grounding in sources, saying so when there is no answer |
| Ticket or email classification | Share of correct labels, which labels get confused with each other |
| Extracting data from documents | Accuracy of each field separately, fields that should stay empty |
| Agent that acts on its own | Task completion, wrong or unnecessary actions, handing back to a person at the right moment |
| Any system | Tone, leaks of personal data, time and cost |
For assistants that work from documents, it pays to keep two measures apart. One is whether the right document was found. The other is whether the right answer was produced from it. Wherever the fault lies, that is where the fix goes. We explain why this split matters in what is RAG?
With agents, behaviour is measured as well as answers. Each step is checked for whether it was needed, and whether the agent asked for approval before anything irreversible. We describe how agents work in what is an AI agent?
How to run human review
Automatic checks catch format; they do not always catch meaning. An answer can be technically correct yet worded in a way you would never send to a customer. Only someone who knows the work will spot that.
To make human review efficient:
- Use a short scoring sheet. Three or four questions are enough: “Is it correct, incomplete or wrong?”, “Does it rest on a source?”, “Would we send it to a customer as it stands?”
- Blind the review. When comparing two versions, the reviewer should not know which answer came from which.
- Use two reviewers. Where two people looking at the same examples disagree, the criterion is unclear. Tighten it there.
- Review little, but often. Reading a random sample every month catches more than one big review a year.
You can also use a language model as the reviewer. At volume it is useful. But that reviewer must itself be measured: on a sample scored by people, check how often the model agrees with them. If agreement is low, its scores cannot be trusted.
What to monitor in production
The test set measures known questions. In production, the system meets something new every day. Monitoring exists to spot what the test set does not cover, early.
The signals we recommend watching:
- Human correction rate. How many of the system’s drafts do people change? If this rises, something has shifted.
- “I don’t know” rate. The share of questions the system could not answer. A sudden rise can mean the sources are out of date, or a new type of question has appeared.
- User feedback. Even a simple “Did this help?” button points the way.
- Time and cost. Response time and model usage. If an agent gets stuck in a loop, this is where it shows first.
- Sample review. Once a week, someone reads a random selection of conversations.
Every new kind of error found in production is added to the test set as a new question. The evaluation grows with the system.
When should you measure again?
Whenever something in the system changes. We recommend re-running the test set when:
- The instructions or examples change
- The model, or the model version, changes
- Source documents are added, removed or updated
- A new tool or permission is given to the system
- A new type of error is noticed in production
Model providers update their versions from time to time and retire old ones. So we recommend recording which model version a system runs on, and not switching without running the test set first.
How to make the go-live decision
An evaluation that does not lead to a decision is only a report. Our recommendation is to agree three thresholds in writing at the start of a project. The numbers differ from project to project — what matters is that they are written down.
| Decision | When it applies | What happens next |
|---|---|---|
| Ready to launch | Thresholds met in critical categories, no errors on irreversible steps | The system opens for use and monitoring begins |
| Supervised use | Overall results are good, but some categories fall below threshold | The system only drafts; a person approves every output |
| Stop | A wrong action, a leak of personal data, or clear errors in a critical category | The fault is fixed, the test set extended, the measurement repeated |
The phrase that matters most in this table is “no errors on irreversible steps”. However high the average, a single error on a payment, a deletion or an outbound message is enough to pull the system back. We set out which steps need human approval in hallucination, privacy and human approval.
One more point. It is tempting to write the success criteria after the system is built, because then they can be fitted to the result. Our view is the other way round: criteria first, system second.
Frequently asked questions
How many test questions does an AI evaluation need?
There is no fixed number. What matters is that every category of question the system will face has enough examples, and that questions with no answer are included too. We suggest starting with a small, well-spread set and growing it as new kinds of question appear in production.
Aren’t the benchmark scores published by model providers enough?
No — those scores come from general-purpose test sets. They know nothing of your customers’ questions, your documents or your business rules. A model that does well on general benchmarks may still struggle with your work.
Is evaluation a one-off job?
It is not. Instructions, documents, model versions and the questions people ask all change over time. That is why we recommend re-running the test set before every change and reviewing samples from production regularly.
Can evaluation be added to an AI system that is already running?
Yes. Real questions are first gathered from the system’s logs, then a test set and criteria are written. The current system’s result on that set becomes the baseline, and every later change is compared against it.
If you are planning an AI system or want to measure one you already have, have a look at our AI systems service, or start with ChatGPT integration for businesses if this is your first integration. To tell us about your project, write to us; we reply within two working days.