Most Teams Can See Their Agents. Few Can Judge Them.

The center of agent engineering has already moved. LangChain's 2026 State of Agent Engineering survey found that 89% of organizations have implemented observability, the layer that lets you see what an agent did, while only 52% run offline evals on a test set and just 37% run online evals on production traffic.¹ We built the mirror but not the yardstick. A team can now auto-trace every tool call and reasoning step, then answer "what did the agent do?" instantly and "was that good?" never.

That gap is not a tooling gap. Evaluation tooling is cheap and mature. It is a discipline gap, and it is exactly where agent proof-of-work goes to die. The same survey reported that 32% of teams name quality concerns as the top barrier to moving an agent into production.² Those teams are not short on traces. They are short on a repeatable way to decide whether a change made the agent better or worse.

Why an agent is not an LLM to evaluate

The instinct is to grade an agent like a model: send a prompt, read the completion, apply an LLM-as-judge. That model breaks for agents in three mechanical ways.³

First, the unit of evaluation is a trace, not a completion. An agent run is a tree of observations: LLM calls, tool executions, retrievals. Each node carries its own quality question, and none of them is visible if you only score the final text. Second, intermediate steps carry independent failure modes. Retrieval can return the wrong documents, a tool can be called with malformed arguments, a loop can repeat the same failing call, and the final answer still reads fine. Third, many agents are conversational, so the session, not the trace, is what the user actually experienced. Five individually reasonable turns can collectively fail to resolve anything.³

The practical consequence is that a correct-looking final answer can hide an unstable, inefficient, or risky execution path. An agent that calls the wrong tool, loops through retrieval it should not have done, and still lands on the right answer is a loaded gun; the same flawed route will fail the next time the context differs by a hair.⁴ When it does, there is no stack trace pointing at where it drifted, because the drift was distributed across a dozen non-deterministic decisions. The observability layer gives you the trace, the raw material this whole discipline grades; we wrote separately about how OpenTelemetry turns agent runs into spans you can inspect. Evals are what turn those spans into a verdict.

Evaluate on more than the final output, and you discover the field standardizes on four dimensions. Trajectory asks whether the agent took a sensible path: step count, unnecessary tool calls, loops and retries. Tool use asks whether it called the right tool with the right arguments, and whether it recovered after a failed call. Task completion asks whether the user's goal was met. Multi-turn quality asks whether performance holds across a conversation, for context retention and goal drift.³ A single aggregate score tells you the agent got worse; a per-dimension score tells you where. Teams serious about shipping agents measure all four, not one number, and the trajectory dimension is where a specific regression type lives, the model update that corrupts one early step and breaks everything downstream, which deserves its own eval treatment in our piece on trajectory regression evals.

What the eval layer is built from

An evaluation for an agent has the same skeleton no matter who builds it. There is a task, which is one test with defined inputs and success criteria. An attempt at a task is a trial, and because model output varies between runs you run several. A grader is logic that scores part of the agent's performance, and a task can carry several graders. The transcript, or trajectory, is the full record of a trial: outputs, tool calls, reasoning, intermediate results. The outcome is the final state of the environment after that trial, not what the agent said. An agent can say "your flight is booked" while no reservation exists in the database, and only the outcome catches that.⁵ This vocabulary matters because teams that skip it end up arguing about "the test" without agreeing on what is being graded.

The dataset is a selection, not a dump

The single highest-leverage decision in an eval program is which tasks you include, and the best source is your own failures. Anthropic's guidance is blunt: start with 20 to 50 simple tasks drawn from real failures, not hundreds you hope you will need.⁵ LangChain's checklist agrees and pushes further, that 20 to 50 hand-reviewed examples you are confident in beat hundreds of synthetic examples nobody verified, and that defining eval tasks is one of the best stress tests of whether your product requirements are concrete enough to build from.⁶

Write unambiguous tasks with a reference solution that proves the task is passable. Vague success criteria become noise in the metrics; two domain experts should independently reach the same pass-fail verdict. And test the negative cases, not just the positive ones. If you only test that the agent searches when it should, you will quietly optimize toward an agent that searches for everything. Anthropic hit this directly building web search evals, needing both "should search" and "should answer from memory" directions to avoid undertriggering and overtriggering at the same time.⁵

Building a good dataset is mostly honest postmortem. Convert user-reported failures into test cases, dogfood the agent daily and turn every error into an eval, pull specific tasks from external benchmarks you adapt rather than running them wholesale, and write focused behavior tests by hand, like "does it parallelize tool calls" or "does it ask a clarifying question when the request is vague."⁶

The harness isolates the agent, not hopes

The harness is the infrastructure that runs evals end to end: it gives the agent its tools, runs trials concurrently, records every step, grades, and aggregates. The rule that matters is that each trial starts from a clean, isolated environment with no shared state between runs. Shared state corrupts the signal in both directions. Leftover files and cached data can cause correlated failures that look like agent regressions but are really infrastructure flakiness, and shared state can inflate performance. Anthropic observed Claude gaining an unfair advantage on some internal tasks by reading the git history from previous trials.⁵ A reference solution also serves here: a 0% pass rate across many trials is most often the signal of a broken task, not an incapable agent, so you double-check the spec before you blame the model.⁵

Graders: prefer code, calibrate the model

There are three families of graders, and the ordering is deliberate: code-based first, model-based where needed, human to calibrate. Code-based graders cover string checks, binary pass-fail tests, static analysis, and outcome verification. They are fast, cheap, objective, and reproducible, at the cost of brittleness to valid variations. Model-based graders handle nuance and open-ended tasks but are non-deterministic and must be calibrated against humans. Human graders give the gold standard and are expensive and slow.⁵

The sharpest advice across every source is also the least followed: grade what the agent produced, not the path it took. There is a strong instinct to assert a specific sequence like "call tool A, then B, then C in that order," and it is the wrong instinct, because agents routinely find valid routes the eval designer never anticipated.⁵⁶ Ask whether the meeting got scheduled correctly, not whether it called check_availability before create_event. And build in partial credit: an agent that identifies the problem and verifies the customer but fails to process the refund is meaningfully better than one that fails immediately.⁵

LLM-as-judge is the practical default at scale because humans cannot sustain the volume. A person can meaningfully review 50 to 100 traces per hour, so at 1,000 requests a day full manual review is 10 to 20 hours daily, which nobody does.⁴ But the judge needs engineering. Give it a way out, an instruction to return "Unknown" when it lacks information, to avoid hallucinated grades. Give it clear structured rubrics, one dimension per isolated judge rather than one judge grading everything. And calibrate it against human labels, starting at around 20 labeled examples and growing toward roughly 100 for production confidence.⁶ Judges drift, so recalibrate on a schedule, and be aware of their known failure modes: verbosity bias, where longer outputs score higher; scoring drift as the model changes; and contradictory results depending on whether outputs are judged individually or side by side.⁴

The grader-matrix decision, taught once, stops most bad evals before they start.

There is a useful line to draw inside the automation: guardrails are not evaluators. A guardrail runs inline during execution, in milliseconds, to block dangerous or malformed output before a user sees it. An evaluator runs after generation, asynchronously, to measure quality and catch regressions. Mixing them, running blocking safety checks as slow asynchronous evals or treating a quality score as a safety gate, gets both wrong.⁶

Read the scores, then distrust them

Metrics are where evals go wrong in public. The two that matter most diverge as the number of trials grows, and teams pick the wrong one by default. Pass@k measures the chance that the agent succeeds at least once in k attempts; as k grows it rises toward certainty. Pass^k measures the chance that all k attempts succeed, and it falls as k grows, because requiring consistency is the harder bar. At k equal to 1 the two are identical, and by k equal to 10 they tell opposite stories, with pass@k approaching 100% while pass^k falls toward zero.⁵ Use pass@k for tools where one success matters and pass^k for an agent where every user expects consistent behavior, and know that choosing the optimistic metric will flatter a flaky system. A 75% per-trial success rate looks fine on paper and means your customer sees failure roughly a quarter of the time, and three in a row only passes 42% of the time.⁵

The deeper trap is trusting scores that were never audited. When Opus 4.5 initially scored 42% on CORE-Bench, an Anthropic researcher found the problem was the eval: rigid grading that penalized "96.12" when the reference expected "96.124991...", ambiguous task specs, and stochastic tasks that were impossible to reproduce. After fixing the graders and loosening the scaffold, the same model scored 95%.⁵ The lesson is not that scoring is broken; it is that a bad grader looks identical to a bad agent until you read the transcripts. Failures should seem fair. When scores do not climb, you need to know whether it is the agent or the eval, which is why "read the transcripts" is the recurring instruction in every serious treatment of the subject.⁵

Close the loop: offline, online, CI

An eval layer that only runs when someone remembers to is not infrastructure; it is a report. The standard architecture closes the loop in both directions.

Offline evals run against a curated dataset with reference outputs before a change ships, and they are regression testing, the safety net that catches a bad prompt, model, or tool change before users do. monday.com built its evaluation strategy on this framing, using offline evals to test groundedness, retrieval accuracy, and tool calling plus edge cases like knowledge base conflicts.⁴ Online evals run on sampled production traffic without reference outputs, catching what a test suite cannot anticipate: unexpected inputs, edge cases you never wrote, and gradual degradation as usage shifts.⁴ Neither works alone, and the production trace is the input to both. A problematic trace caught online becomes the dataset item that reproduces it offline, and that item stays as a permanent regression test.⁴⁷

Wire the whole thing into CI. Run an experiment against a versioned regression dataset on every pull request that changes prompts, models, or tool schemas, and fail the pipeline when scores miss a threshold, exactly like a test suite. Keep that CI dataset small enough to run on every PR, tens to low hundreds of items, and run the larger sweeps on a schedule or before release.⁷ For online judging, start at a 10% sampling rate to control judge-model cost, and trigger the expensive LLM-based checks only when the cheap deterministic checks drop.⁴

Set the bar before you read results. Separate capability evals, which start at a low pass rate and give you a hill to climb, from regression evals, which sit near 100% and protect what already works. Without the separation you either stop improving because you are only guarding existing behavior or you ship regressions because you are only chasing new capability.⁵⁶ And watch for saturation: an eval at 100% catches regressions but gives no signal for improvement, and as scores near the ceiling, real capability gains show up as tiny score bumps. SWE-bench Verified moved from about 40% to above 80% in roughly a year, and frontier models are now bumping the ceiling, the point where the eval stops separating them.⁵

This eval gate is the front half of a larger release discipline. Judging whether a change is good, which this whole post is about, only matters if you can then ship the good version without breaking the bad one, and that is the version, canary, and rollback problem we covered in agent release engineering. The eval decides yes or no; the release engineering decides how the "yes" gets out safely.

It is the same discipline we apply inside our own delivery pipeline: any change that alters agent behavior, a prompt, a model, a tool schema, or orchestration logic, gets judged against a fixed regression set before it moves forward, not after a user reports the decline. You do not need our specifics. The pattern is the point: opinionated evaluation gates, on every change, with the failures feeding the next regression case.

A minimum stack you can ship this week

Enough is known now to avoid analysis paralysis. The smallest defensible setup, per the current field guidance, is a golden dataset of 50 to 100 representative trajectories, three rubrics covering reasoning, tool correctness, and final answer quality, a single judge model, and a second judge model for cross-checks if the budget allows.⁸ Calibrate that judge against roughly 100 human-labeled cases, run it offline before every change, sample a slice of production afterward, and turn every confirmed failure back into a test case.

The honest version of the current state, from LangChain's survey, is that the industry has more observability than evaluation by a wide margin, 89% against 52% and 37%.¹ That is not a reason to wait. It is the reason the eval layer is the highest-leverage infrastructure decision left, because it is the layer that converts traces, which every team already has, into whether the next model upgrade or prompt tweak is safe to release. The teams ahead are the ones who built the yardstick when it still felt optional.