Measure a Workflow, Not a System

Published 2026-10-11

Evaluating LLM Agents, part 2. Series page and full outline.

I don't think "how good is our agent?" is a question that can be answered as posed. "How often does the agent complete this workflow, and how does it fail when it doesn't?" can be, and this part is about the difference between the two.

What a workflow is

A workflow has three parts, and none of them is the agent:

1. A distribution of starting situations. The tickets, repositories, documents, or requests the agent will face, in the proportions it will face them.
2. An environment. The tools, data, and state the agent reads and changes while it works.
3. A definition of done, stated in terms of the end state of the environment rather than in terms of what the agent says.

The system (a model plus a harness) is what runs the workflow. I find it useful to think of the workflow as the yardstick: you hold it still, change the system, and compare the numbers you get.

Figure 1. Light boxes are the workflow and stay fixed. The dark box is the system, which is what you change.

In notation, let $x \sim \mathcal{D}$ be a starting situation, $E$ the environment, and $S$ the system. A run produces a trajectory $\tau$ and an end state $e_T$. The workflow score is

$$\mathbb{E}_{x \sim \mathcal{D}}\;\mathbb{E}_{\tau \sim \mathrm{Run}(S, E, x)}\big[\, s(x, e_T, \tau) \,\big],$$

where $s$ is the definition of done. The rest of the series is about estimating this quantity cheaply and without fooling yourself, which turns out to be most of the difficulty.

Why system scores fail

Scores reported for a system in general, rather than for a workflow, fail in three ways that I keep running into.

They do not transfer. SWE-bench draws its issues from twelve Python repositories (Jimenez et al., 2024), so a score on it describes that workflow and says nothing about whether the same system will process a refund correctly. Kapoor et al. (2024) describe this as a conflation of model developers' needs with downstream developers' needs: a benchmark that ranks models is not an evaluation of a product.

They are confounded. A reported number is a property of a model, a harness, an environment, and a task set together. SWE-agent's authors showed that interface design alone moves an agent's SWE-bench score, by ablating the agent-computer interface with the model held fixed (Yang et al., 2024), and the Holistic Agent Leaderboard analyzes models, scaffolds, and benchmarks as three separate dimensions (Kapoor et al., 2025). A score with any of those unrecorded can't be reproduced, and in my experience usually isn't.

They can be gamed. Many agent benchmarks have inadequate holdout sets or none, and agents overfit to them through shortcuts (Kapoor et al., 2024). Log inspection on the Holistic Agent Leaderboard found agents searching Hugging Face for the benchmark instead of solving the task.

Done is a property of the end state

τ-bench grades a customer-service conversation by comparing the database at the end against an annotated goal state (Yao et al., 2024). WebArena checks the resulting page and backend state rather than the agent's narration (Zhou et al., 2023). Anthropic's example is a booking agent that "might say 'Your flight has been booked'... but the outcome is whether a reservation exists in the environment's SQL database" (Anthropic, 2026).

The way I write a definition of done, it has two lists: the changes that have to have happened, and the changes that must not have. AgentRewardBench had experts annotate web-agent trajectories for success, side effects, and repetitiveness separately, because a trajectory can reach the goal while doing something it should not have (Lù et al., 2025). Both lists should exist before you look at a single run, because once you've seen the runs it gets very hard not to grade toward them.

Where the starting situations come from

The starting situations in an eval set should be drawn from, or shaped like, production traffic, and stratified by scenario so that a regression in one scenario is visible rather than averaged away (Husain, 2024). The set will keep growing from observed failures: Shankar et al. (2024) found that people cannot state their grading criteria until they have graded outputs, which they call criteria drift. Part 6 covers how to build the set.

Component metrics are diagnostics

Retriever recall, tool-call accuracy, and judge agreement with humans are all worth measuring, as explanations for movements in the workflow score rather than as substitutes for it. When a component metric improves and the workflow score does not, that component was not the bottleneck. AgentBoard's progress-rate metric is a good example of a diagnostic: it reports how far along a task an agent got, alongside whether it finished (Ma et al., 2024).

Anthropic splits suites into capability evals, which start with a low pass rate and give a team something to improve, and regression evals, which should stay near 100% (Anthropic, 2026). Both run over the same workflow, and a capability task that becomes reliable moves to the regression suite.

Offline and online

An offline eval is a sample from the workflow, run in a controlled environment. Online measurement is the workflow itself, with real users and real consequences. Husain (2024) orders the levels by cost: assertions on every change, model and human grading on a cadence, A/B tests after significant changes. Offline results are a prediction, online results are the confirmation, and each online failure should be added to the offline set. Part 5 covers what makes offline evaluation of agents possible, and Part 11 covers online evaluation for systems that act.

Tenets

• The unit of evaluation is a workflow: a distribution of starts, an environment, and a definition of done.
• Done is a property of the end state, with a list of required changes and a list of forbidden ones.
• Hold the workflow fixed, change the system, and record model, harness, and environment versions with every number.
• Draw starts from production and grow the set from failures.
• Component metrics explain the workflow score rather than replacing it.

References

Jimenez, C. E. et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR.
Kapoor, S. et al. (2024). AI Agents That Matter.
Yang, J. et al. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS.
Kapoor, S. et al. (2025). Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation.
Yao, S. et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.
Zhou, S. et al. (2023). WebArena: A Realistic Web Environment for Building Autonomous Agents. ICLR 2024.
Anthropic (2026). Demystifying evals for AI agents. Anthropic Engineering.
Lù, X. H. et al. (2025). AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories.
Husain, H. (2024). Your AI Product Needs Evals.
Shankar, S. et al. (2024). Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. UIST.
Ma, C. et al. (2024). AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents. NeurIPS.