The Harness Tax: Same Coding Model, Up To Five Times The Cost, Barely Any Difference In Success

Two new studies have arrived at the same conclusion from different directions, and together they question a purchase decision many teams have already made. The harness around a coding agent, its system prompt, tools and context management, barely changes how often the agent succeeds. It changes the bill enormously.

The Berkeley HarnessTax study evaluated 21 model and harness combinations across seven models and three harnesses on SWE-bench Lite and Terminal-Bench 2.0, and found that the choice of coding-agent scaffold moved success rates by a couple of percentage points while multiplying inference costs up to fivefold. A second study from researchers at Beihang University, Wuhan University and Singapore Management University, accepted at ISSTA 2026, added the controlled version: with the model fixed and one variable at a time changed, a heavyweight harness and a minimalist one produced statistically equivalent results on hundreds of tasks, at costs that differed by as much as threefold.

This article covers both studies, the mechanism behind the cost difference, the vendor benchmark problem they expose, and what a team actually buying a coding tool should do. Everything comes from the papers and the study pages, read directly.

Study one: HarnessTax, and the price of the default

The HarnessTax project, from UC Berkeley's Sky Computing Lab with collaborators, asked a simple question: could another harness help the same model solve more tasks or reduce costs? The setup was systematic. Seven models, three scaffolds, Claude Code, Codex CLI and the open-source Pi, 21 model-scaffold pairs, evaluated on SWE-bench Lite and Terminal-Bench 2.0.

Three findings came out, and each one cuts against how these tools are usually bought.

Scaffold choice barely moved success rates. The average harness effect stayed within plus or minus two percent on SWE-bench Lite and about plus or minus five percent on Terminal-Bench 2.0. Claude Fable 5, the strongest model tested, solved 97.8 percent of attempts in Claude Code, 96.7 percent in Codex and 96.7 percent in Pi.

Cost moved enormously. Claude Fable 5's runs in the heavyweight scaffold cost about twice its Pi runs, 1.33 dollars against 0.67, for a 1.1 percent success-rate increase. Across shared models, Claude Code cost about 2.0 times Pi and 1.6 times Codex on SWE-bench Lite by geometric mean, and 1.5 times Pi on Terminal-Bench 2.0. The authors name the phenomenon directly: paying extra for essentially the same quality because of a different harness is a harness tax, and you may be paying it whenever you accept a coding agent's default harness without comparing alternatives.

A minimal scaffold was competitive. Pi, an open-source coding agent exposing just four tools, read, write, edit and bash, reached the Pareto frontier on both benchmarks. The same model took nearly identical numbers of turns under Pi and under Claude Code, 15.4 against 15.3 turns per attempt for one model, while the heavyweight cost twice as much. The difference begins with the first model call: across all seven models, Claude Code's initial context averaged more than ten times Pi's, from longer instructions and larger tool schemas.

The finding that got the least attention and may matter most: provider optimisation does not guarantee the best pairing. Across the six Anthropic and OpenAI models and both benchmarks, an alternative harness achieved the highest observed success rate in nine of twelve comparisons, including cases where a vendor's own harness underperformed a competitor's on the vendor's own model. The study's framing is dry and devastating: a shared provider does not guarantee the best pairing.

Study two: the controlled experiment, and the noise floor

The second study, accepted at ISSTA 2026 with authors from Beihang University, Wuhan University and Singapore Management University, examined a different question: how much does code execution inside the agent loop actually contribute, and what does it cost?

The design isolates one variable. The scaffold stays fixed, Claude Code with one model, Codex with another, and the open-source OpenCode with a third, and only execution access varies across four paradigms, from prohibited through quota-limited to unrestricted. The evaluation ran 3,000 end-to-end repair attempts across 200 SWE-bench instances, on top of an analysis of 7,745 public agent traces from leaderboard submissions.

The headline result is uncomfortable for the agentic default. On the commercial agents, the resolve-rate gap between prohibited and unrestricted execution was 1.25 percentage points, not statistically significant. On the open-source agent it was roughly zero. Prohibiting execution saved 56 to 62 percent of tokens and 48 to 54 percent of wall-clock time on one commercial agent, and eliminated the need to maintain per-repository test environments entirely.

The study also quantified the noise floor, which is the part that most vendor benchmarks ignore. Rerunning the exact same setup produces score variation: a cited separate study found spreads of 2.2 to 6.0 percentage points across ten identical reruns on SWE-bench Verified. The researchers computed what that means for evaluation design: a 45-task suite can only catch a 13-point gap about half the time, and even 447 tasks can only resolve a 5.2-point difference at the observed task-level variation. Any vendor claim of a 2 to 3 point harness gain from a single-run comparison is inside the noise.

Their audit of their own early runs is the detail that deserves the widest circulation. In early web-enabled testing, they discovered that 312 of 439 trials by two model families, 71 percent, had retrieved the original human-written fix from GitHub. They discarded those runs and blocked network access in the final evaluation. A benchmark result can look like capability and be contamination, and the only defence is controlling the network.

Why vendor benchmarks mislead, in both directions

Both studies spend real effort on a meta-problem: the numbers teams use to choose coding tools are harder to read than they look.

The controlled study identifies two distortions. First, many published comparisons run each configuration once, while identical reruns of the same setup can vary by several percentage points on their own. A vendor chart showing a 2.4 point improvement over a competitor, produced from single runs, cannot be distinguished from rerun noise without a rerun of the baseline. Second, the harnesses being compared are often not on equal footing: the paper cites a study in which the same agent scored 19.1 percent with a minimal benchmark adapter and 73.4 percent with the full adapter, on the same model, which means the adapter, not the model or the agent's design, decided the result.

Test conditions can distort things further. The 71 percent contamination finding came from the researchers auditing their own early runs after noticing suspiciously high scores with web access enabled. Any evaluation that lets agents reach the internet during testing needs the same audit, because retrieval of the answer key reads exactly like capability from the scoreboard's point of view.

The Berkeley study adds the sourcing problem from the other side: public leaderboards often compare coding agents running different scaffolds, so the reader cannot isolate what the scaffold contributed even when the numbers are honest. Both papers arrive at the same practical rule from different directions. Rerun the baseline, control the network, and treat single-run deltas under five points as unmeasured.

What this means for teams that self-host

One consequence of the pairing result is easy to miss. If the scaffold barely moves success rates, then the choice of coding agent becomes largely a cost, privacy and integration decision, which changes the calculus for self-hosting.

An open-weight model served on your own infrastructure, driven by a minimal open-source harness with four tools, is now a defensible configuration by the evidence of these studies, not a compromise. The Berkeley study showed the minimalist reaching the Pareto frontier on both benchmarks; the controlled study showed the open-source scaffold statistically equivalent to the commercial ones. For teams with data-residency requirements, or budgets measured in self-hosted GPU hours rather than API bills, that combination is worth evaluating seriously rather than defaulting to the commercial bundle.

The same logic applies to the cheap API tier. The controlled study tested an open-source scaffold with a 32-billion-parameter code model and found identical results with and without execution access, at a third of the token cost. Teams building high-volume internal tooling on inexpensive models have less reason than ever to pay the heavyweight tax, because the studies find its benefits concentrated rather than general.

None of this makes heavyweight scaffolds useless. Long-horizon work, complex codebases and tasks at the edge of a model's ability are exactly where the Berkeley study expects richer scaffolds to earn their keep, and the authors say so. The claim these studies refute is narrower and more damaging: that the default heavyweight bundle is automatically worth its multiple on ordinary work.

The mechanism: the tax starts with the first token

Both studies locate the cost difference in the same place, and it is not intelligence.

The initial context that a harness sends on the first call, the system prompt plus tool schemas, is sent again with every step of the loop, because agents re-send the conversation transcript each turn. In the second study's measurement of three harnesses, the first call carried 16,581 tokens in the heavyweight, 7,025 in a mid-weight, and 829 in the minimalist one. The researchers' own summary is precise: what separates the harnesses' per-step weight is their own text, not the repository's. The Berkeley study found the same ordering at larger scale, with the heavyweight harness's initial context averaging more than ten times the minimalist one's.

Even prompt caching does not erase the gap. In the second study, 96 to 99 percent of input tokens in these loops hit the provider's cache, and repricing the tokens across different vendor cache rates scaled the bill without changing which harness was most expensive. The cached-input economics we covered in the inference caching guide reduce the penalty without removing the ranking.

One more nuance from the controlled study complicates any simple cost-cutting recipe. Turning off reasoning saved money for one model and collapsed another's success rate by 11 points, while producing changes inside the run-to-run noise for two others. The effect of reasoning budgets depends on the model and harness combination together, which is the same conclusion the model routing article reaches about boundaries between tiers: the pairing is the unit of evaluation, not either half alone.

What this means for a team buying a coding tool

Four actions follow, and the first one costs an afternoon.

Benchmark the pair, not the product. The unit that matters is model plus harness on your own tasks, measured twice. Both studies recommend rerunning baseline configurations, because single-run differences of a few points are indistinguishable from luck. A two-hour internal eval on twenty of your own tasks, run twice, tells you more than any leaderboard.

Question the default pairing. The Berkeley result that an alternative harness won nine of twelve provider-paired comparisons means the bundled combination is not automatically the best one. Your Claude models may not need the harness they ship with, in the study's own memorable phrasing, and neither may anyone else's models.

Attack the first call. The per-step weight of a harness is its own text. Leaner tool schemas and shorter standing instructions reduce the tax on every subsequent turn. The controlled study even found that removing a heavyweight harness's dedicated file-editing tools, forcing the model to edit through shell commands, left success within the range of the control runs.

Audit network access in evaluations. The 71 percent contamination finding is a warning that applies to any benchmark with web-enabled agents. If your evaluation cannot explain where its answers came from, it is not measuring what you think it is.

There is also a research-culture point worth making. Both studies arrive at a conclusion the vendor marketing runs opposite to: harness complexity should be treated as an empirical trade-off, as the second study puts it, not as a feature list. The same scepticism applies to the self-improving harnesses we covered in the Harness2 article and the control-loop research in the meta-reasoning piece: improvements that cost more executions per task have to earn their price on your workload, not on a benchmark table.

Where the numbers stop

• The HarnessTax results are limited to the two benchmarks its authors name, on tasks the models may have encountered during training, as the study itself notes. Results may differ on other benchmarks and workloads, in the authors' own words.
• The controlled study's restricted-execution findings apply to the bug classes in the 200 SWE-bench instances studied. The authors state the open question directly: execution is not inherently unhelpful, its benefit is concentrated on specific instances, and how an agent should decide when to invest in execution remains unresolved.
• The cost figures use published list prices at the time of each study. Provider pricing changes, and cached-input rates differ by vendor, so absolute dollar figures age faster than the ratios.
• Neither study is independently replicated as far as I could find, though the second corroborates the first, which is a partial check.
• I read the HarnessTax blog and study page in full and the ISSTA paper's abstract, introduction, background and experimental-setup sections. I did not read the full results and analysis sections of the paper, so figures beyond those quoted here are not repeated.

FAQ

What is the harness tax?

The harness tax is the difference in inference cost between coding-agent harnesses running the same model on the same tasks, when success rates are nearly identical. The Berkeley HarnessTax study measured the same model costing up to five times as much across harnesses on its benchmarks, with success rates moving within plus or minus two to five percent, and traced the gap mainly to the harness's own text: the system prompt and tool schemas resent with every step.

Does a more elaborate harness make a coding agent more capable?

On the evidence of these two studies, not reliably. The controlled study found that heavyweight and minimalist harnesses were statistically equivalent on hundreds of program-repair tasks with the model held fixed, and the Berkeley study found success-rate differences within a few percentage points across three harnesses. Elaborate harnesses can still help in specific settings, and the controlled study found execution's benefit concentrated on particular instances, but neither supports choosing a harness on benchmark deltas of a few points from single runs.

How should a team evaluate coding agents?

Benchmark the model and harness pair on your own tasks, run each configuration at least twice, control network access during evaluation so agents cannot retrieve answers, and measure cost per task alongside success rate. The studies also suggest auditing the first call: the harness's initial context size predicts much of the cost difference, and it is the easiest thing to compare across tools before committing to a full evaluation.

The short version

Two studies from different teams converge on the same finding: the harness around a coding model barely changes how often it succeeds while multiplying what it costs. Berkeley's HarnessTax measured 21 model-harness pairs and found success-rate effects within a few points while costs spread up to fivefold, with a minimal four-tool open-source harness competitive on both benchmarks it tested. A controlled ISSTA study held the model fixed, varied only execution access, and found statistically equivalent results at up to threefold cost differences, with 56 to 62 percent token savings from restricting execution. The mechanism is the harness's own text, resent on every turn, starting with a first call that averaged more than ten times the weight in the heavyweight harness. The practical response is to benchmark the pair on your own tasks at least twice, question the default pairing rather than accepting it, slim the first call, and control network access in any evaluation, because the researchers caught 71 percent of their own early trials retrieving answers from the web.

Sources

• Arena and UC Berkeley Sky Computing Lab, HarnessTax: How Much Does the Harness Matter for Coding Agents?, covering the 21 model-harness pairs, the success-rate stability, the cost ratios and the provider-pairing findings: https://arena.ai/blog/coding-agents-harness-tax
• Lin, Zhu, Zhou, Wang, Sun, Yang, Lo and Li, an empirical study of code execution in LLM-based program repair, accepted at ISSTA 2026, covering the 7,745 trace analysis, the 3,000 controlled repair attempts, the noise-floor measurements and the contamination audit: https://arxiv.org/html/2606.26978v1