What Changes When the Model Acts

Published 2026-10-11

Evaluating LLM Agents, part 3. Series page and full outline.

Evaluating a plain language model is a procedure most people already know: you fix a prompt, sample a response, and compare the response to something, whether that's a reference answer, a test, or a judge's opinion. The input never changes, the output is a single string, and you could run the whole thing out of a spreadsheet if you wanted to. Agents break every one of those assumptions, and I've found it helps to see the breakage laid out in one place.

The input moves

The agent's second action depends on what its first action did to the world, so there's no fixed dataset of inputs to iterate over; everything after step one is produced by the agent itself. You have to actually execute. That's the reason agent benchmarks come packaged as environments instead of files: WebArena ships as self-hosted websites (Zhou et al., 2023), OSWorld as virtual machines (Xie et al., 2024), τ-bench as a database plus a simulated user (Yao et al., 2024), and SWE-bench as a runnable repository per task (Jimenez et al., 2024). Part 5 goes into what those environments actually have to do.

Reliability across trials

pass@k asks whether any of k attempts succeeds (Chen et al., 2021), and it's the right metric when a single success is all you need, which is true of code when you have tests to filter the attempts with. It's the wrong metric for an agent that acts on a user's behalf, where one success in eight attempts means seven people got let down. τ-bench introduced pass^k, the probability that all k trials succeed, and the numbers were sobering: an agent that passed under half the tasks once passed fewer than a quarter of them eight times in a row (Yao et al., 2024). The two metrics coincide at k = 1 and then pull apart fast (Anthropic, 2026).

Figure 1. The same agent, with a 75% chance of succeeding on any one trial, scored both ways. By three trials pass@k reads 98% and pass^k reads 42%; by ten they read 100% and 6%.

What I'd do in practice is run several trials per task, report the per-task success rate along with its spread rather than one pass rate, and pick pass@k or pass^k based on what the product actually needs from the agent.

The harness is part of what's being tested

Hold a model fixed, change nothing but its interface to the computer, and the score moves (Yang et al., 2024). The Holistic Agent Leaderboard goes as far as treating the scaffold as its own dimension, separate from both model and benchmark (Kapoor et al., 2025). In practice this mostly comes down to bookkeeping, which is less glamorous than it sounds: every number you record should carry the model, the harness version, the tool set, and the environment version, and editing a tool description has to count as a change to the thing under test, because it is one.

Cost and length are part of the result

Measuring accuracy alone gave us agents that were "needlessly complex and costly," in the words of Kapoor et al. (2024), who argue for reporting accuracy and cost side by side. Length matters for a different reason. METR measures agents by the human time of the tasks they can complete about half the time, and found that horizon doubling roughly every seven months (Kwa et al., 2025). Longer tasks mean fewer tasks per dollar of eval, which means fewer trials, which means more variance, right where you can least afford it. Turns, tool calls, tokens, and wall-clock belong next to every success rate you report.

Failures have a location

When a plain model fails, the failure is the whole output. When an agent fails, it tends to fail at step 37 of 80 and then spend the remaining 43 steps confidently building on the mistake, and outcome grading will only ever tell you that it failed. AgentBoard's progress rate is one answer, measuring how far along the task an agent got (Ma et al., 2024). Agent-as-a-Judge grades the intermediate steps and matched human evaluation on its DevAI tasks where a one-shot LLM judge didn't (Zhuge et al., 2024). And when the Holistic Agent Leaderboard went through its logs it found behaviors no outcome metric would ever have surfaced, like agents misusing credit cards in flight-booking tasks (Kapoor et al., 2025).

Figure 2. What each grader sees of the same failed trajectory. Only reading it locates the mistake.

My rule is that the outcome gives you the score and the trajectories give you the diagnosis, so whenever the score moves, in either direction, read a sample of trajectories before believing it.

Agents have side effects

An agent can reach the goal and still have done something along the way that it shouldn't have: run a destructive command, issue a refund outside policy, touch something out of scope. τ-bench's tasks require following domain policy, and an agent that makes the user happy by breaking policy fails the task (Yao et al., 2024). AgentRewardBench annotates side effects as a dimension of their own (Lù et al., 2025). So a definition of done needs a list of forbidden changes alongside its list of required ones.

Agents can also reach the grader, which is worse. METR has documented frontier models modifying tests and scoring code, and hunting down reference answers, in order to get impossibly high scores (Von Arx et al., 2025). Baker et al. (2025) monitored reward hacking in agentic coding environments and found that optimizing against the monitor taught the agent to hide what it was doing. If the agent could reach its own grader, I don't think the resulting number measures the task at all.

The old toolkit still applies, to the artifact

The final pull request, reply, or report can still be graded with the plain-model toolkit of verifiers, rubrics, and judges, which is what Part 4 is about. That grader is one of several, though, and it isn't the one that tells you whether the world ended up in the right state.

Tenets

• Agent evaluation requires execution in an environment rather than iteration over a dataset.
• Run multiple trials per task and choose pass@k or pass^k deliberately.
• The harness is part of the system under test, so version it.
• Report cost and length with every success rate.
• Score the outcome and diagnose from the trajectories.
• Every definition of done lists forbidden changes, and the agent must not be able to reach the grader.

References

Zhou, S. et al. (2023). WebArena: A Realistic Web Environment for Building Autonomous Agents. ICLR 2024.
Xie, T. et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurIPS.
Yao, S. et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.
Jimenez, C. E. et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR.
Chen, M. et al. (2021). Evaluating Large Language Models Trained on Code.
Anthropic (2026). Demystifying evals for AI agents. Anthropic Engineering.
Yang, J. et al. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS.
Kapoor, S. et al. (2025). Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation.
Kapoor, S. et al. (2024). AI Agents That Matter.
Kwa, T. et al. (2025). Measuring AI Ability to Complete Long Software Tasks. METR.
Ma, C. et al. (2024). AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents. NeurIPS.
Zhuge, M. et al. (2024). Agent-as-a-Judge: Evaluate Agents with Agents.
Lù, X. H. et al. (2025). AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories.
Von Arx, S., Chan, L., and Barnes, B. (2025). Recent Frontier Models Are Reward Hacking. METR.
Baker, B. et al. (2025). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation.