How to Test an AI Agent Before You Ship It (AI Evals, Plainly)
The fastest way to test an AI agent is to stop trusting your gut and build a small eval set: ten to twenty real examples, each with an input, the context the agent gets, an expected result, and a rule that grades it pass or fail. Run that set every time you touch the prompt, model, or tools. The score tells you, objectively, whether your change helped or quietly broke something. That is the whole discipline.
TL;DR
- "It worked when I tried it" is not testing. It is a coincidence with good lighting.
- An eval set is to an AI agent what unit tests are to code: a repeatable pass-or-fail score on real examples.
- Start with 10 to 20 cases pulled from actual tasks and real failures, not hundreds of synthetic ones.
- Score with the cheapest method that fits: exact match, keyword check, an LLM judge, or a human spot-check.
- Run the set on every change. The first time it catches a regression you would have shipped, it pays for itself.
Why "it worked when I tried it" is not testing
Most agents are shipped on vibes. You build the thing, type in three questions, the answers look great, and you push it. Then a real user phrases something slightly differently, or you upgrade the model, or you tweak one line of the system prompt, and the whole thing starts confidently doing the wrong job. You do not find out until someone complains.
The problem is that an LLM is nondeterministic and absurdly sensitive. The same prompt can give a clean answer and a broken one on consecutive runs. A word you added to "improve" the system prompt can wreck an unrelated task you never thought to recheck. Three hand-typed examples cannot see any of that. They tell you the agent can succeed, not that it reliably does.
Testing replaces "can it work" with "how often does it work, and did my last change make that number go up or down." That shift, from a hopeful demo to a tracked score, is the entire point of evals.
Build a small eval set first
An eval set is just a list of test cases. One case is the smallest unit of testing: a single task with a way to judge the result. Get the shape of one case right and the rest is repetition.
Where do the cases come from? Three places. Your most common real tasks, so the everyday path is locked down. Your scariest edge cases, the inputs where a wrong answer would actually hurt. And every bug a user has ever hit, turned into a permanent case so it can never silently come back. Ten to twenty of those beats two hundred synthetic prompts you generated to feel thorough and then never read.
Keep them in the dullest format that works: a JSON file, a spreadsheet, rows in a table. The format does not matter. Running them on every change is what matters.
How to score an AI agent's output
Scoring is where people freeze, because "was this answer good" feels subjective. It is less subjective than it looks. Pick the cheapest method that fits the task, and reserve the expensive ones for the answers that genuinely need them.
- Exact match. The output must equal a known value. Perfect for classifications, extracted fields, yes/no calls, or a tool name. Fast, free, zero ambiguity. Use it whenever you can.
- Keyword or rule check. The output must contain (or must never contain) certain strings, must be valid JSON, must match a pattern. Great for format, required disclaimers, and banned content.
- LLM-as-judge. A second model reads the input, the agent's answer, and a clear rubric, then returns pass or fail with a reason. This is how you grade open-ended writing, summaries, and explanations at scale. Calibrate the judge against a batch of human-graded examples first, then spot-check it, because a lazy judge will pass things it should not.
- Human spot-check. You read a small sample by hand. Slow and unscalable, so use it to calibrate the automated checks, not as your main scorer.
The trap is grading "did this sound smart." Score the outcome you actually care about: correct, correctly formatted, right tool called, on policy. A confident, eloquent, wrong answer should fail, and a blunt correct one should pass.
Run it as a loop, not a launch
Testing an agent is not a gate you clear once before launch. It is a loop you live in for as long as the agent is alive. Every prompt edit, model swap, and new tool runs the set again before it goes out.
The loop compounds. Each real failure becomes a permanent case, so the same bug can never ship twice. Over a few months your set quietly turns into a precise map of every way your agent has ever broken, and your pass rate becomes a number you can actually defend to yourself. If you are still building the agent itself, the how to build an AI agent walkthrough pairs cleanly with this loop.
Catch regressions before your users do
A regression is when a change you made to fix or improve one thing silently breaks another. Without a test set, you find regressions in production, from angry messages. With one, you find them in the thirty seconds it takes the set to run.
This is the single highest-value reason to keep evals. The day you upgrade to a newer model to get better reasoning, your set might show that the new model is also worse at following your output format, or more likely to skip a tool call. You see that before you ship, weigh it, and decide on purpose instead of getting surprised. The model upgrade that "obviously" helps is not always a clean win, and a five-minute eval run is how you know.
Pair the pre-ship set with a small post-ship check. Running a couple of canary cases against live traffic right after a deploy catches the failures that only show up in the real world. That post-deploy monitoring habit is exactly what The Canary Method lays out, and it is the natural other half of an eval set.
A 30-minute starter you can run today
You do not need a platform or a framework to begin. You need a file and thirty minutes.
- Open a file and write 10 cases. Pull five from the tasks your agent does most, three from inputs that scare you, and two from bugs you remember. For each, jot the input and one line on what a good answer must include.
- Run the agent on all 10 and paste the outputs in. Resist fixing anything yet. You are taking a baseline.
- Score each one pass or fail against your "must include" line. Be strict. A near-miss is a fail.
- Write the score at the top: 7 of 10. That number is now your truth. Every future change has to keep it or beat it.
- Fix your worst failure, then re-run all 10. Confirm the fix worked and that you did not knock down a case that was passing. Welcome to the loop.
That is a working eval set. It is crude, it is honest, and it already puts you ahead of nearly everyone shipping agents on vibes. You can graduate to a real eval tool later, once the discipline is a habit.
If you want the practical AI-for-builders playbook in your inbox, including the tools and workflows behind this, join the newsletter. It is where the tactical stuff lands first.
The bottom line
Testing an AI agent is not a fancy ML problem. It is the boring, decisive habit of writing down what good looks like and checking it on every change. Build ten honest cases, score them with the cheapest method that fits, run the set as a loop, and let it grow one real failure at a time. The first regression it catches before your users do will make you wonder how you ever shipped without it. Trade the vibe-check for a score, and ship like you mean it.
How do you test an AI agent?
You build a small set of real examples, run the agent on each one, and score the output against what a good answer should look like. Each example is an eval case: an input, the context the agent gets, an expected result, and a rule for grading it. You start with ten to twenty cases drawn from real tasks, run them every time you change a prompt, model, or tool, and add a new case whenever you find a failure. The point is to replace one-off vibe-checks with a repeatable pass-or-fail score you can trust.
What is an AI eval?
An eval, short for evaluation, is a test for an AI system. One eval case is a single input plus a way to judge whether the output was acceptable. An eval set is a collection of those cases you run together to get a score, like 17 of 20 passed. Evals are to AI agents what unit tests are to normal code: they tell you, objectively, whether a change made things better or worse instead of leaving you to guess from a couple of lucky examples.
How do you measure if an AI agent is good?
Pick a small number of outcomes that actually matter for your use case, such as factual accuracy, correct format, calling the right tool, or staying on policy, and score each one on every run. Track the pass rate over time. A good agent is not one that impressed you once; it is one that passes your eval set consistently and does not regress when you change the prompt or upgrade the model. Measure the outcome you care about, not how clever the answer sounds.
Can you use an LLM to grade an AI agent's output?
Yes, and for open-ended answers it is often the only practical option. This is called LLM-as-judge: you give a second model the input, the agent's output, and a clear rubric, then ask it to score pass or fail with a reason. It scales far better than reading every answer by hand. The catch is that the judge can be wrong or lenient, so you calibrate it against a batch of human-graded examples first and spot-check it regularly.
How many test cases does an AI agent need?
Start with ten to twenty real cases covering your most common tasks and your scariest edge cases, not hundreds of synthetic ones. A small, honest set you run on every change beats a huge set you never look at. Grow it by adding a case every time a user hits a bug or the agent does something dumb, so the set hardens around real failures over time.
OpusJake is Jake Schincariol's operating system for building with AI: agents, workflows, prompts, and the free resources behind them. Get the next move every week.