Agent evals start with error analysis: read a sample of real traces, label what actually went wrong, count the failure modes, and fix the biggest one. Hamel Husain and Shreya Shankar teach this method in the most popular AI evals course going, and most teams never need to build an LLM judge to get value from it.
Contents
Why I keep sending people the same two names
Every AI engagement we run hits the same wall. The demo works. The pilot gets sign-off. Then it goes in front of real users, quality wobbles, and nobody can say whether the last prompt change made things better or worse. The team starts arguing from anecdotes - someone's screenshot of a bad answer versus someone else's screenshot of a good one.
At that point I stop talking and send two links. Hamel Husain and Shreya Shankar run the AI evals course that most of the industry has now been through, and their public writing is the clearest treatment of this problem I have found. Husain is an ML engineer with a long history at Airbnb and GitHub; Shankar did her PhD at UC Berkeley on exactly this question of how you evaluate systems built on unreliable components. Between them they have written down the method, rather than the vibes.
The method is unglamorous and it works. Sample real traces from your agent. Read them. Write a free-form note on each one where something went wrong - not a score, a sentence describing what happened. Once you have a pile of notes, group them into categories and count. You now have a ranked list of failure modes with numbers attached, and the argument about what to fix next is over.
Error analysis is most of the value
The thing that surprises people is how much of the benefit arrives before you have automated anything.
The first time a team sits down and reads fifty of their own traces properly, they find things nobody suspected. A retrieval step that silently returns nothing on a whole class of queries. A tool the agent calls with the right name and the wrong argument shape. A system prompt instruction that two other instructions quietly contradict. None of that shows up in an aggregate score. It shows up when a human who understands the domain reads the transcript.
That is also why this work cannot be fully delegated to a model. Husain and Shankar are blunt about it: an AI can help you sort and cluster once you know what you are looking for, but the initial pass - deciding what counts as a failure in your domain - is the part that is important. Getting a domain expert to do that pass, rather than an engineer guessing at what the domain expert would say, is where we get the highest signal.
New session with @sh_reya on How to Automate Evals with AI (correctly)
— Hamel Husain (@HamelHusain) July 6, 2026
The most important part of the eval workflow is finding issues. Shreya does a live demo of how to steer the AI iteratively to find unknown unknowns
Chapter summaries:
00:00:00 Introduction
00:00:55 Why… pic.twitter.com/VuQR4qh5ml
You probably don't need an LLM judge
This is the part I find myself repeating most often, so I will say it plainly. Most teams do not need LLM-as-judge, and reaching for it early is usually a way of avoiding the reading.
An LLM judge is a model you have to evaluate before you can trust it to evaluate anything else. Shankar's UIST paper on aligning LLM-assisted evaluation makes the recursion explicit - a generated evaluator inherits every weakness of the model that generated it, so you end up needing human-labelled examples anyway to check the judge agrees with you. If you were always going to need the labelled examples, you may as well start there.
Evals are arguably the hardest part of LLMOps. LLMs mess up, so we check them w/ other LLMs, but this feels icky. Who validates the validators??
— Shreya Shankar (@sh_reya) April 22, 2024
We built an interface to align LLM-based evals with user preferences, learning a lot about why this is hard: https://t.co/g7UXuBznv9
In practice, once you have your failure taxonomy, a large share of the top failure modes turn out to be checkable in ordinary code. Did the output parse? Did the agent call the tool with valid arguments? Did the retrieved chunk contain the identifier the user asked about? Did it refuse when it should have answered? Those are assertions, they run in CI, they cost nothing, and they never drift.
And for many failures, you fix it once and it no longer needs evaluation, any evaluation would pass 100% of the time - the definition of a waste of tokens!
Save the judge for what is left: the genuinely subjective failure modes that are frequent enough to matter and that you now have labelled data for. Build it in that order and it works. Build it first and you have added a second unreliable system to debug alongside the first.
The short reading list
If you read nothing else, read these in this order.
- A Field Guide to Rapidly Improving AI Products - Husain's end-to-end account of the loop, including why a decent data viewer is the best tooling investment most teams can make.
- In Defense of AI Evals, for Everyone - Shankar's argument for why this is a general product skill, not a specialist ML one.
- Using LLM-as-a-Judge For Evaluation: A Complete Guide - the right way to build a judge, for when you actually need one.
- LLM Evals: Everything You Need to Know - the FAQ they wrote from thousands of student questions. Skim it, then come back when you hit a specific problem.
- AI Evals For Engineers & PMs - the course itself, if you want the structured version with assignments.
For how this fits the wider picture of what actually slows AI teams down, my notes from Melbourne's AI Engineering Unconference cover the same ground from the organisational side.
Frequently Asked Questions
- Error analysis is the practice of sampling real traces from your AI system, writing open-ended notes describing what went wrong in each one, then grouping those notes into a counted taxonomy of failure modes. The count is what turns quality arguments into a prioritised backlog.
- Start with a sample you will actually finish - 30 to 100 traces is enough to surface the dominant failure modes in most systems. You are looking for the shape of the distribution, not a statistically rigorous estimate. Sample again after each round of fixes.
- Usually not, at least not first. Build deterministic assertions for the failure modes that code can check, and only build a judge for subjective failure modes that are frequent enough to justify the effort and for which you already have human-labelled examples to align against.
- Traditional tests assert a fixed expected output. Evals measure the rate of failure modes across a set of realistic inputs, because a non-deterministic system will never produce one correct string. In practice you want both: assertions for the things with a right answer, and measured failure rates for the things without.
- A domain expert should do the labelling, with an engineer alongside to handle the tooling. The value comes from someone who knows what wrong means in your context. Engineers guessing at domain judgments is the most common way this exercise produces a useless taxonomy.
- As soon as you have real usage to sample, which is usually earlier than teams think. You do not need a framework, a platform, or a budget line to start - a spreadsheet of traces and a column for notes is a legitimate v1 and will out-perform a tool nobody has read the output of.
- Yes, and they matter more for agents. A multi-step agent gives you a trace with several places to fail, so error analysis has to identify which step broke rather than only that the final answer was wrong. The method is the same; the unit of analysis moves from the response to the trajectory.
Get Started
We'll help you build it.
About Freshwater Futures
We build evals into every AI product engagement we run, because it is the difference between a demo and something you can put in front of customers. If your team has an AI feature that works in the demo and wobbles in production, get in touch - a first round of error analysis usually takes a day and changes what the roadmap says.
