The Environment Is the Eval

Published 2026-10-11

Evaluating LLM Agents, part 5. Series page and full outline.

For a plain language model, the eval is a dataset. For an agent, I've come to think the eval is the environment: somewhere the agent's actions have consequences that you can reset, observe, and check. Every earlier part quietly assumed one, since a state-based definition of done needs a state to read, multiple trials need a world that starts the same way each time, and a verifier needs something to verify against. This part is about what the environment has to do for all of that to work.

What an environment must do

Reset

Every trial has to start from the same state, which sounds obvious until you try to guarantee it. SWE-bench ships an executable repository per task (Jimenez et al., 2024), τ-bench resets its database between conversations (Yao et al., 2024), and WebArena hosts its own copies of web applications so the sites do not change under the agent (Zhou et al., 2023). Without a reset there's no such thing as a second trial of the same task, and so no pass^k either.

Isolate

The agent should be able to reach what it needs and nothing else, and in particular it shouldn't be able to reach the grader. Filesystem and network boundaries are the two that matter: Anthropic's sandboxing for Claude Code is built on those two and reports cutting permission prompts by 84% in internal use (Anthropic, 2025), and microVMs such as Firecracker provide the same boundaries at close to container cost (Agache et al., 2020). Isolation is usually discussed as a safety measure, but it is also a measurement requirement, since an agent that can read the scoring code produces a number about the scoring code (Von Arx et al., 2025).

Expose state

The grader must be able to read the final state: database rows, files, URLs, interface properties. OSWorld's evaluation scripts inspect the file system, application configuration, database contents, and interface elements after the agent stops (Xie et al., 2024). A definition of done that cannot be read out of the environment is not usable yet.

Simulate the other party

Support, sales, and coaching agents talk to someone. τ-bench uses a model to play the user (Yao et al., 2024), and τ²-bench constrains that simulated user with tools and observable state to improve fidelity, showing large performance drops once the user can also act on the world (Barres et al., 2025). The simulator's mistakes become the eval's noise, so they should be measured separately, as τ²-bench does by separating reasoning errors from communication errors. Part 10 covers simulator design.

Resemble production

An offline environment is a model of the world, and like any model it drifts. Online-Mind2Web evaluated web agents on 300 tasks across 136 live websites and found earlier results over-optimistic, a gap it attributes to the benchmarks (Xue et al., 2025). The environment should be refreshed from production traces on a schedule, and a widening gap between offline and online numbers should be treated as a bug in the environment.

Run cheaply, many times

The Holistic Agent Leaderboard's 21,730 rollouts cost about 40,000 USD and needed hundreds of VMs (Kapoor et al., 2025). Cost limits the number of trials, and the number of trials sets the variance. Reusable infrastructure exists: Inspect from the UK AI Security Institute (UK AISI), BrowserGym for web agents (Le Sellier De Chezelles et al., 2024), Terminal-Bench's harness for shell tasks (Terminal-Bench Team, 2025), and SWE-ReX for running agent commands in sandboxes at scale (SWE-agent, 2025).

Figure 1. Production traces become tasks in the environment. The environment grades harness changes and rewards weight updates, and both ship back to production, which produces new traces.

Where the two hills meet

Part 1 described two hills: weights, which have a gradient, and harnesses, which have only an eval. The environment serves both.

SWE-Gym built 2,438 executable task instances and used them to train agents, gaining up to 19 points absolute on SWE-bench (Pan et al., 2024). R2E-Gym generated more than 8,700 tasks procedurally from commits (Jain et al., 2025). SWE-smith turned any Python repository into an environment and produced 50,000 instances from 128 repositories (Yang et al., 2025). Reasoning Gym did the same for procedurally generated, verifiable reasoning tasks (Stojanovski et al., 2025). The verifier that grades a harness in these systems is also the reward that trains the weights.

Holdouts matter more as a result. Once an environment is a training signal, a public copy of it is contaminated for evaluation. SWE-bench Pro, from Scale AI (where I work), splits into a public set, a held-out set, and a commercial set of proprietary repositories, and reports on the sets that agents cannot have seen (Deng et al., 2025). The same applies to a team's own environment: some slice of it should never be optimized against, by a person or by a gradient.

Offline and online

Offline evaluation happens in the environment, which makes it fast, repeatable, and safe, and also blind to anything the environment doesn't model. Online evaluation happens in production, through A/B tests (Kohavi, Tang, and Xu, 2020), shadow runs where the new system acts without its actions taking effect, and plain user feedback. Online results are the ground truth, but they're slow, they're noisy, and for a system that acts they're risky; you can't A/B test refunds casually.

The loop in Figure 1 is the one I'd recommend: iterate offline, validate online, turn each online failure into an offline task, and refresh the environment from each new batch of production traces. Husain (2024) describes this cycle for products, and Anthropic describes the same combination of evals, production monitoring, and A/B tests for Claude Code (Anthropic, 2026).

Across the five parts

The eval does the gradient's job on the harness hill, so it deserves the care normally given to a loss function. Its unit is a workflow: starting situations, an environment, and a definition of done expressed as state. Agents add execution, multiple trials, cost accounting, trajectory reading, and a list of forbidden changes. Verifiers apply where a program can decide, rubrics where experts can agree on a checklist, and pairwise preference otherwise, with the verifier itself measured. The environment makes all of this possible, serves training and evaluation alike, and should keep a held-out slice.

Seven more parts are outlined on the series page, starting with building the eval set and calibrating the judge.

References

Jimenez, C. E. et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR.
Yao, S. et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.
Zhou, S. et al. (2023). WebArena: A Realistic Web Environment for Building Autonomous Agents. ICLR 2024.
Anthropic (2025). Beyond permission prompts: making Claude Code more secure and autonomous. Anthropic Engineering.
Agache, A. et al. (2020). Firecracker: Lightweight Virtualization for Serverless Applications. NSDI.
Von Arx, S., Chan, L., and Barnes, B. (2025). Recent Frontier Models Are Reward Hacking. METR.
Xie, T. et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurIPS.
Barres, V. et al. (2025). τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment.
Xue, T. et al. (2025). An Illusion of Progress? Assessing the Current State of Web Agents.
Kapoor, S. et al. (2025). Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation.
UK AI Security Institute. Inspect: An open-source framework for large language model evaluations.
Le Sellier De Chezelles, T. et al. (2024). The BrowserGym Ecosystem for Web Agent Research.
Terminal-Bench Team (2025). Terminal-Bench.
SWE-agent (2025). SWE-ReX: Sandboxed code execution for AI agents. Code.
Pan, J. et al. (2024). Training Software Engineering Agents and Verifiers with SWE-Gym.
Jain, N. et al. (2025). R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents.
Yang, J. et al. (2025). SWE-smith: Scaling Data for Software Engineering Agents.
Stojanovski, Z. et al. (2025). Reasoning Gym: Reasoning Environments for Reinforcement Learning with Verifiable Rewards.
Deng, X. et al. (2025). SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? Scale AI.
Kohavi, R., Tang, D., and Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.
Husain, H. (2024). Your AI Product Needs Evals.
Anthropic (2026). Demystifying evals for AI agents. Anthropic Engineering.