They're not something you build once. They have to follow your agent from synthetic benchmarks, to production traces, to changing requirements.
Ask a team how they evaluate their AI agent and you'll usually hear some version of: "We have a test set. Our pass rate went from 70% to 90%." Then the agent meets real users, and that number stops meaning much.
The best advice in this field is still Hamel Husain's:[look at your data]. Read traces, find the failures, and write evals for the errors you actually see. I agree with all of it. But for agents, looking at the data is necessary and not enough, because agents add something most eval setups ignore: everything keeps moving. The users change, the agent changes, and the definition of "correct" changes.
The core mistake is treating agent evals as a fixed test set: build it once, climb the pass rate, ship.
Here are the three places where that breaks.
Your evals are built for users who don't exist, and real users leave you one trace you can't replay
Before launch, you have no real traffic. Your evals are synthetic or hand-curated, which means they reflect your team's imagination of what users will do. Climbing a pass rate on that set is risky. You're tuning the agent to an imagined user, and real users show up with different phrasing, messier goals and edge cases nobody wrote down.
Pre-launch evals are still useful, but as hypotheses about where the agent might fail, not as a scoreboard. Favor coverage and realism over pass rate, keep some cases you never tune against, and plan to replace them with evals built from real traces.
After launch, the traces arrive, and with them a new problem. A production failure is a single observation of a stochastic system, in a state you usually can't recreate. The customer, the data and the tool responses won't line up the same way again.
The common reaction is to save the transcript as a "golden example" and ask an LLM whether the new answer looks similar. That doesn't test the agent. The agent will never see that exact sequence again, and text similarity says almost nothing about whether it did the right thing.
Turning a failure into an eval means rebuilding the situation, not copying the conversation:
- The environment: the data state the agent acted on, and its tools, or faithful stand-ins for them.
- The user: a simulated user with the original goal and knowledge, and no hint of the right answer.
- The trigger: whatever made it hard. That might be an ambiguous request, a missing verification step, or two requests in one message.
Then decide how to measure success:
- Grade the outcome, not the prose. If the agent refunded the wrong order, check the database with code. Use an LLM judge only for what code can't check, such as tone or whether an explanation is correct. Give each judge one criterion, and verify that it can fail and that it agrees with a human.
- Report a rate, not a verdict. Run the scenario several times and report how often it fails. A single pass from a stochastic agent proves very little, and "too few runs to tell" is an honest result.
- Test the opposite case too. If you teach the agent to stop transferring customers to a human too early, add a case where transferring is right. Otherwise you'll get an agent that never transfers.
Your "successful" traces are hiding failures
Teams read the traces that failed. The ones that "worked" get the least attention: no error was thrown, the task was marked complete, and the user didn't complain.
But agents fail quietly. Some patterns I've seen in support agents:
- It says "Let me look that up for you" and then ends its turn without looking anything up.
- It makes a change on an account before verifying who it's talking to.
- It hands the customer to a human and drops the two other requests they made.
- It promises a follow-up that never happens.
Every one of these reads as polite and competent, and every one would pass a casual review.
This isn't only a human blind spot. In a recent study,Parlance Labsgave 100 production traces to six automated reviewers: three eval platforms and three coding agents. The reviewers recovered 74–87% of the failures a domain expert had found, and each surfaced new issues the expert had missed. But every one of them missed failures where the trace looks fine and the product's goal still isn't met: the agent gave up at a sales objection, sent Markdown symbols in an SMS, or kept a conversation that should have gone to a human.
What helps:
- Read traces against what the agent is supposed to do, not just for errors. "Did it achieve the goal under our policies?" is a different question from "did anything break?"
- Track near misses and risks, not just failures. A risky pattern that worked out fine this time is an eval waiting to be written.
- Use automation to triage, and people to judge. Let tools sort thousands of traces. Then have a domain expert read a sample deeply and make the call on anything ambiguous.
Requirements change, and your evals quietly rot
Agents live inside products, and products change. A refund policy gets stricter, a new feature launches, or a "never do X" becomes "X is fine for enterprise customers". Every eval you wrote encodes yesterday's idea of "correct", and almost nobody tracks which ones still hold.
There's a deeper problem underneath: most teams don't have a single source of truth for how the agent should behave. Requirements live in a policy doc, a few Slack threads, the system prompt and someone's head. That has always been a gap in software development, and with agents it decides whether your evals mean anything. You can't tell if an eval is still valid when nobody can say what "correct" currently is.
The usual result is an append-only test suite that drifts out of date in two ways. Some evals now block good changes because they still expect old behavior. Others pass wrong behavior because the bar moved and nobody updated them. Six months in, nobody remembers why half the evals exist.
What helps:
- Keep one living list of your agent's critical requirements. Store it in the repo, next to the code, as plain files. Have your coding agent maintain it: update it when policies, prompts or product decisions change, and flag when code and requirements disagree. It becomes the reference every eval points to.
- Link every eval to the requirement it tests. If you can't name the requirement, the eval probably shouldn't exist.
- When a requirement changes, review every eval linked to it. Decide whether to keep it, update it or retire it, and record why. A sentence that's still in the policy word for word doesn't mean the eval is still valid if an exception was added next to it.
- Never change an eval while you're using it to compare two versions of the agent. Freeze it, compare, and only then repair it. After any change, measure again before claiming an improvement, because scores from an older version of the eval can't be compared with the new one.
- Retire evals that always pass. They cost money and attention and tell you nothing new.
Evals are a living system
The teams that get agent evals right don't have a bigger test set. They have a process: synthetic cases give way to real traces, failures become realistic and repeatable scenarios, "successful" runs get audited against requirements, and every eval stays tied to an up-to-date list of what the agent is supposed to do.
A checklist you can use this week:
- Treat pre-launch evals as hypotheses, and plan to replace them with evals built from real traces.
- When a production failure matters, rebuild the situation (environment, user and trigger) instead of saving the transcript.
- Grade outcomes with code, run each scenario several times, and validate every LLM judge against human labels.
- Read a sample of yoursuccessfultraces against your requirements.
- Keep a living list of your agent's requirements (your coding agent can maintain it), link every eval to one, and review the linked evals whenever a requirement changes.
We've spent the past year building for exactly these problems, and we'll share more soon. In the meantime, I'd like to hear which one your team runs into most.
