How do you test an AI agent before it goes live?
An AI agent that looks good in a demo can still break your ERP on day one. How I test agents against real past cases before they touch production.
The demo always works. Someone feeds the agent a clean example invoice, it reads supplier, amount and due date, and everyone in the room nods. Then it goes live, and the third invoice is a scanned PDF with a handwritten correction, and the agent books it to the wrong cost centre.
Nothing was wrong with the model. Nobody had checked what it does with the cases that actually come in.
Here’s how I test an agent before it gets anywhere near a production system.
Start with cases you already solved
The best test material is your own history. Every company that runs a process has a pile of finished cases: invoices that were booked, emails that were answered, claims that were assessed. For each one you know what a person decided and why.
I pick a set of those, usually a few dozen to start, and run the agent on them. Then I compare its result with what your people did. Not “does it sound plausible”, but “did it come to the same booking, the same answer, the same value”.
That set is often called a golden set. The name is a bit grand. It’s a folder of real cases with the right answer next to each.
Pick the awkward ones on purpose
A test set full of clean examples tells you the agent handles clean examples. You knew that from the demo.
The cases worth having are the ones your team grumbles about. The supplier who sends three invoices in one PDF. The customer who writes in two languages. The claim where the product doesn’t exist anymore and someone had to find the successor model. If the agent fails those, you want to know now, not in week two.
I ask the people who do the work: “Which cases would you never trust a new colleague with?” Those go in first.
Decide what counts as good enough
An agent will not get every case right. Neither does a new employee. The question is which mistakes you can live with.
A wrong field in a draft that a person reviews anyway is cheap. A wrong amount that goes straight into the ledger is not. So before testing, I agree with the customer where the agent may act alone and where a person checks. That decides how strict the test has to be.
Run it next to your people first
When the numbers on past cases look good, I’d still not let the agent write to production yet. Let it run in shadow mode first: it sees the real incoming cases, produces its result, and a person compares it with their own decision. Nothing the agent does reaches the ERP or the customer yet.
That tends to turn up case types nobody thought of. They go into the test set, and the agent gets fixed for them.
Test again after every change
This is the step people skip. A new model version, a reworded prompt, a changed business rule: any of them can shift results in places you didn’t touch. I rerun the full set of past cases after every change and compare it with the last run before anything goes live.
In one project, an agent for an expert assessment firm proposes replacement values for damaged devices with sources. Every change to it is replayed against earlier cases first. The expert still checks and decides each case, but the replay shows what a change did to results before it reaches daily work.
What this looks like in practice
- A set of real past cases with the right result, including the awkward ones
- A clear line between what the agent does alone and what a person reviews
- Shadow mode before it writes anything
- A log of every run, so you can see why it decided what it did
- The same test set rerun after every change
None of this is exotic. It’s mostly the discipline of writing down what “right” means before you build. If you want to talk through how that would look for one of your processes, my AI automation work starts exactly there.
Short answers
How do you test an AI agent before connecting it to production systems?
Run it against a set of real past cases where you already know the right outcome, compare its result with what your people decided, and only connect it to live systems once the error rate is acceptable. Then let it run in shadow mode next to the people for a while before it writes anything itself.
How many test cases does an AI agent need?
Enough to cover the normal case and the awkward ones. For most business processes that is somewhere between a few dozen and a few hundred past cases. The awkward ones matter more than the count.
Do you need to test again when the AI model changes?
Yes. A new model version, a changed prompt or a new business rule can all change results. Rerun the same set of past cases after every change and compare before it goes live.
