← Blog

Before you build an agent, benchmark the simple version

2026-09-29

The first design for a new agentic workflow tends to arrive as a diagram: a planner, a researcher or two, a critic, a synthesizer, arrows between them. Drawing it is the fun part. The question that should come first is duller. How well does one strong model with the right inputs already do on the task, and what does each box in the diagram add to that number?

I wrote in July about how to build an eval harness for LLM agents, and in August about how many runs a comparison needs. Those posts cover how to measure an agent. This one is about what to measure first, before the architecture exists, and what recent controlled studies say about whether the extra agents earn their cost.

My short answer: define success, build a small eval, and measure the simplest system that can fairly attempt the task. Then add components one at a time, each aimed at a failure the eval showed you, and keep a component only if the eval says it helped.

A control group for agent architecture

This is older advice than agents. Google’s Rules of Machine Learning opens with “Don’t be afraid to launch a product without machine learning,” puts metrics second, and makes Rule #4 “Keep the first model simple and get the infrastructure right.” The reason given for a simple first model is the one that applies here: “Your simple model provides you with baseline metrics and a baseline behavior that you can use to test more complex models.”

An agent is a larger version of the same problem. The thing being scored is a configuration of model, prompt, tools, retrieval, loop, retries, and orchestration, and any of those can move the number. Without a control, a 90% score from a five-agent system says nothing about the five agents.

The vendors who sell the models say the same. OpenAI’s practical guide to building agents recommends building the prototype “with the most capable model for every task to establish a performance baseline,” then trying smaller models, and states “Our general recommendation is to maximize a single agent’s capabilities first.” Its evaluation best practices page is more direct: “The decision to use a multi-agent architecture should be driven by your evals. Starting with a multi-agent architecture adds unnecessary complexity that can slow down your time to production.” Anthropic’s Building effective agents, which I discussed in what engineers review in agentic systems, makes the same recommendation from the other direction.

The eval does not need to be large. Anthropic’s guide to agent evals says “20-50 simple tasks drawn from real failures is a great start,” because early changes tend to have large effects: “this large effect size means small sample sizes suffice.” Writing those tasks also forces the team to agree on what the product is supposed to do, which is often the first thing the exercise exposes.

A plain model can be an unfair baseline

“Benchmark a vanilla LLM first” is close to right, and the gap matters. Take an assistant that answers engineering questions from a library of component datasheets. A model with no retrieval will fail most of those questions because it has never seen the datasheets. That result is a capability floor. It tells you how much the model knows unaided, which is worth recording, but it is a poor control for a multi-agent retrieval system, because the two differ in the information they receive as well as in their architecture.

The control that isolates the architecture is a matched baseline: one model with the same datasheet index, the same tools, the same answer format, and the same budget as the system you plan to build. Comparing a planner-researcher-critic pipeline against that baseline measures what the orchestration buys. Comparing it against the bare model mostly measures retrieval.

Budget is the variable most often left unmatched. A five-agent system that spends five times the tokens of a single agent has received more test-time compute, and more compute alone can raise scores. Tran and Kiela held reasoning-token budgets equal across Qwen3, DeepSeek-R1-Distill-Llama, and Gemini 2.5 and found that single agents “consistently match or outperform” multi-agent systems on multi-hop reasoning. They also found that API-based budget controls, particularly Gemini 2.5’s, did not enforce budgets reliably, which can inflate apparent multi-agent gains. Their paper is a preprint, but the methodological point stands independently: count tokens and cost per task in every arm.

The rungs from rules to multi-agent

I find it useful to name the configurations in order of complexity and measure upward until the eval stops improving:

B0 and B1 are often skipped, and B0 is where some projects end. A task that a regex, a SQL query, or a small classifier handles at 99% does not need a model. B2 is usually the matched baseline for everything above it.

Each rung is a separate configuration run against the same task set, with task success, cost per task, latency, and the full trace recorded for each. The eval harness post describes the machinery. The part specific to this workflow is that every rung stays in the results table after you move past it.

What matched comparisons have found

Several 2026 studies ran the matched comparison directly, and the results lean toward the simpler configurations.

BenchAgent. Fu and colleagues built BenchAgent to run single-agent and multi-agent workflows under “the same benchmark loader, tool access, answer contract, usage accounting, and trajectory logging,” using GPT-4.1 across ten reasoning, coding, and tool-use benchmarks. At most one of six multi-agent systems beat the matched single-agent anchor on balanced average accuracy. That one, EvoAgent, led by 1.44 points, inside the paper’s one-run uncertainty band. The other five trailed by 2.56 to 11.29 points and cost more.

Single agents replaying workflows. Xu and colleagues tested seven benchmarks covering coding, math, question answering, domain reasoning, planning, and tool use, and found that “a single agent can reach the performance of homogeneous workflows with an efficiency advantage from KV cache reuse.” Homogeneous here means every role in the workflow runs on the same model. Their result says less about workflows that mix different models.

Scaling studies. Kim and colleagues at Google evaluated 260 configurations across six agentic benchmarks, five architectures, and three model families. The effect of adding agents ranged from +80.8% on decomposable financial reasoning to −70.0% on sequential planning, and gains shrank as the single-agent baseline got stronger. The journal version appeared in July 2026 in Nature Machine Intelligence under a title that summarizes the finding: Capable language models can outgrow the benefits of collaboration.

The pattern predates 2026. Agentless (Xia et al., FSE 2025) replaced an autonomous coding agent with a fixed three-phase pipeline of localization, repair, and patch validation. On SWE-bench Lite it solved 32.00% of issues at $0.70 each, the best performance and lowest cost among the open-source agents it was compared with in mid-2024.

Where more agents paid for themselves

The same literature has clear wins for more complex designs, and they share a feature: the added structure supplies something one context window cannot.

Anthropic’s multi-agent research system, with a Claude Opus 4 lead and Claude Sonnet 4 subagents, outperformed a single Opus 4 agent by 90.2% on Anthropic’s internal research eval. The mechanism is parallel breadth: many independent searches, each in its own context. The cost is tokens. Anthropic reports that agents use about 4 times the tokens of chat and multi-agent systems about 15 times, and that on BrowseComp token usage alone explained 80% of the performance variance. They also note that “most coding tasks involve fewer truly parallelizable tasks than research,” so the design does not transfer to every domain.

A Mount Sinai study in npj Health Systems (Klang et al., March 2026) gave batches of 5 to 80 mixed clinical tasks (retrieval, extraction, dosing) either to one agent handling everything or to an orchestrator assigning each task to a dedicated worker. Multi-agent accuracy went from 90.6% at 5 tasks to 65.3% at 80. Single-agent accuracy fell from 73.1% to 16.6%. The multi-agent runs used up to 65 times fewer tokens. The mechanism is context isolation: one agent carrying 80 unrelated tasks in one context degrades much faster than workers that each hold one task.

BenchAgent has a win of this kind too. In a separate GAIA study, a Claude-Code-style runtime workflow reached 66.72% overall and 69.23% on Level 3, more than 20 points above the strongest non-Claude baseline. The authors decline to credit any one mechanism: runtime delegation, persistent artifacts, verifier-stage control, and context management “remain confounded,” and isolating the cause “requires controlled ablations.”

So the useful question for a new task is whether its structure resembles these cases. Breadth-first search over independent subtopics, large heterogeneous batches, and work that overflows one context are candidates for decomposition. Tightly coupled reasoning, where each step depends on the last, is where the matched studies found single agents holding their own.

Each component is a hypothesis about a failure

Anthropic’s harness design post puts it in one sentence: “Every component in a harness encodes an assumption about what the model can’t do on its own, and those assumptions are worth stress testing.” That turns each box in the diagram into a claim the eval can check:

If the baseline traces do not show the failure, the component has nothing to fix. If they do, add that one component, rerun the same tasks, and compare success, cost, and latency against the rung below. OpenAI’s guide names two concrete triggers for splitting one agent into several: prompts with “many conditional statements,” and tool overload, where “some implementations successfully manage more than 15 well-defined, distinct tools while others struggle with fewer than 10 overlapping tools.” Both are observable in traces before any restructuring.

The benchmark is also a measurement

A small eval built early will have defects, and it will steer design decisions anyway, so it is worth knowing how evals go wrong.

Grading can be wrong. The Agentic Benchmark Checklist (Zhu et al., NeurIPS 2025) found that τ-bench counted empty responses as successes and that problems of this kind can misstate agent performance by up to 100% in relative terms. The infrastructure can move the score: Anthropic measured a 6-point gap on Terminal-Bench 2.0 between its most- and least-resourced container setups, with the model held fixed. One run can mislead: τ-bench found GPT-4o succeeding on under 50% of tasks and on under 25% of retail tasks when required to pass the same task eight times, and Rabanser and colleagues report that “recent capability gains have only yielded small improvements in reliability.”

Two habits keep the early eval honest. First, keep the comparison sizes in view. At 20 to 50 tasks you can see a jump from 40% to 80%, and you cannot see a difference between 88% and 91%; the runs post works through the arithmetic. Second, split the tasks once the suite grows. A development set you tune against daily becomes training data after a few dozen prompt revisions. Keep a regression set of past failures and a holdout set you run rarely, and add real production cases as they arrive.

Anthropic’s eval guide also notes that “teams can get surprisingly far through a combination of manual testing, dogfooding, and intuition.” A spreadsheet of 20 cases scored by hand counts as a first eval. What matters is that the simple configuration gets scored on the same cases as the complex one.

Rerun the baseline when the model changes

A component that fixed a real failure can stop earning its place. In the harness design post, Claude Sonnet 4.5 showed “context anxiety” on long tasks, so the harness reset the context periodically. Claude Opus 4.5 “largely removed that behavior on its own,” and the author “was able to drop context resets from this harness entirely.”

That only shows up if the simpler configuration is still in the eval. When a new model arrives, run the old model with the full design, the new model with the full design, and the new model with the simple design. If the last two are close, the difference is the cost of maintaining components that no longer fix anything. I made a related argument about deterministic tools in the agentic RF design stack post: the parts of a system worth building carefully are the ones that would stay useful if models stopped improving.

Limits of this evidence

Most of the 2026 studies cited here are preprints; Kim et al. and Klang et al. are the peer-reviewed exceptions. They use different models, budgets, and task families, so their numbers do not combine into one effect size. BenchAgent’s controlled comparison used one model, GPT-4.1, and one run per configuration. Anthropic’s 90.2% comes from an internal eval that outsiders cannot reproduce. The matched-baseline studies mostly cover reasoning, coding, and tool-use benchmarks, and a production workflow with long-running state or human handoffs may behave differently.

The studies above include clear multi-agent wins. Their benefit depends on the task, and the way to learn which kind of task you have is to measure a simpler system on it.

The working rule

Write down what success means, score the simplest system that can fairly attempt the task, and add one component at a time against a failure you can see in the traces. Keep the simple configuration in the results table for as long as the system exists.

This is an independent analysis written on my own time, using the public sources cited above. The views are my own and do not represent any current or former employer.