Evaluating LLM Agents

A series on measuring agentic systems, written for engineers who ship agents without touching weights, and for researchers who train them. The starting point is that most teams can only change the harness around a model, the harness has no gradient, and the eval has to do the gradient's job.

Each part opens with a note on what it covers and what it leaves to a later part. The outline below lists the parts that are written and the ones that are not. Corrections and disagreements are welcome by email.

Outline

  1. Two Hills: Weights and Harnesses. RLHF, DPO, and RLVR climb the weights with a scalar signal. Harnesses have no signal but the eval. Why the second hill is new, and who is on it.
  2. Measure a Workflow, Not a System. The unit of evaluation is a distribution of starts, an environment, and a definition of done expressed as state. Why system-level scores do not transfer.
  3. What Changes When the Model Acts. Moving inputs, pass^k versus pass@k, the harness under test, cost as part of the result, failures with a location, and agents that reach the grader.
  4. Verifiers First, Rubrics Second. Verifiable versus non-verifiable tasks, when a rubric is the wrong tool, why verifiers need measuring too, judge hygiene, and a worked example with a judge calibrated against 50 human labels.
  5. The Environment Is the Eval. What an environment must do, where training and evaluation share it, and the offline-online loop.
  6. Building the Eval Set from Production Traces (planned). Sampling and stratification, open-coded error analysis, criteria drift in practice, synthetic tasks, and maintaining a holdout.
  7. Calibrating a Judge (planned). The judge as a classifier against human labels, true positive and true negative rates, error-corrected pass rates, how many labels and from whom, tuning the judge prompt on one split and testing on another, and when to re-calibrate.
  8. Statistics for Small Eval Sets (planned). Error bars, paired comparisons between system versions, trials versus tasks, intervals on pass^k, and deciding whether a change regressed.
  9. Trajectory Grading and Failure Taxonomies (planned). Failure modes with a location, trajectory assertions, forbidden actions, and tool-call checks.
  10. Simulated Users (planned). Designing user simulators, measuring their noise, and dual-control settings.
  11. Online Evaluation for Systems That Act (planned). Shadow runs, A/B tests with side effects, escalation and override rates, and closing the loop back to the offline set.
  12. Long-Horizon and Multi-Agent Systems (planned). Cost accounting over long tasks, credit assignment across agents, and evaluating orchestration.

Notes

Links go to arXiv or the original source. New parts are listed on the writing page.