Agent evals should explain why, not just score what.

An eval should tell us more than whether a run was good.

A useful agent eval answers three questions: Did the agent produce a good result? Why did the run behave that way? Did a specific change make it better or worse?

A final score answers only the first question. Agents are systems made of models, prompts, context, memory, tools, state, and infrastructure. A weak output could come from a poor plan, missing context, the wrong tool, bad parameters, a failed recovery, or a judge that misunderstood the task.

Put LLM and VLM judges at the center.

The evaluation factors describe what matters. LLM and VLM judges do the grading. Each judge receives the candidate run, the relevant evidence, and a grounded reference package.

Cost and latency are measured alongside these judgments. They are not substitutes for quality, but they show whether an improvement is operationally worthwhile.

Ground the judges in real work, then scale with synthetic data.

Hand-written scenarios are useful, but they rarely capture the variety of real usage. The strongest starting point is approved, de-identified production-shaped data: actual goals, realistic state, long conversations, common tool combinations, and known failure modes.

Models can expand those examples into a larger evaluation set. They can create paraphrases, edge cases, longer conversations, expected tool results, successful traces, strong reference artifacts, and deliberately flawed outputs. This gives the judge both positive examples and hard negatives.

A reference is not always one perfect answer

Creative and open-ended tasks may have many valid outputs. For those tasks, use a reference package: required constraints, accepted examples, disallowed failures, expected state changes, and a clear rubric. The judge grades against the package without forcing the candidate to copy one surface form.

Record enough of the run to explain the grade.

The judge should see the evidence needed for the factor it is evaluating. An outcome judge needs the request, reference package, and rendered artifact. A tool judge needs the intended subgoal, tool call, result, and resulting state. A context judge needs the information available to the agent at that point in time.

Production-derived records need consent or approved use, de-identification, access controls, limited retention, and a deletion path. Raw hidden chain-of-thought is not required; observable actions, concise decision summaries, state transitions, and tool evidence are more useful and safer to store.

Prefer relative comparison before absolute scores.

It is difficult for a judge to invent a reliable score for one output in isolation. It is usually easier to ask a focused question: for the same scenario and reference package, is candidate A better, worse, or equivalent to candidate B?

Show the baseline and candidate in randomized order and hide which system produced each one. Ask for a decision on each factor, require evidence, and report win, loss, and tie rates. Deterministic checks—such as schema validity, required elements, state invariants, and exact constraints—should run before model judgment.

A scalar score can still be useful for dashboards, but it should be derived from calibrated factor-level decisions rather than treated as the source of truth.

Evaluate the judges before trusting their grades.

Create a reviewed calibration set containing clear wins, clear failures, close calls, and ties. Compare the judge's decisions with human preferences for each factor, not only the overall result.

The evaluation pipeline also needs normal software tests. Filters, defaults, fixtures, joins, and schemas can fail while still producing believable numbers. Run an unchanged scenario several times to measure the noise floor before treating a small movement as real.

Turn judge preferences into a reliable release signal.

Map every scenario to the capabilities it exercises—such as planning, memory, a specific tool, or visual output. Compare a candidate with a versioned baseline while changing one variable at a time. This makes the result easier to attribute.

A gate should consider factor-level win rates, important hard failures, judge confidence, and normal variance. It should also report tokens, cost, latency, tool calls, and retries. A cheaper run is only better if it maintains the required quality.

Automated blocking comes last. First establish trustworthy references, complete evidence, calibrated judges, stable baselines, and clear ownership for approving exceptions.

Build the system in the order that makes each result trustworthy.