Even when they use the same model, coding AI tools can differ in cost and success rate depending on the environment that runs them. In a study published on October 3, 2026, Yangze Liu and Zhongyi Han of Shandong University compared three types of execution infrastructure and examined how much that infrastructure affects an AI's ability to solve problems. On 45 difficult code-fixing tasks, the share of tasks whose outcomes flipped when the harness was changed was about the same as when the same configuration was simply rerun. Meanwhile, with GLM-5.3-Flash, the average cost of a single attempt differed by roughly a factor of three.
The three compared systems were Claude Code, OpenCode, and mini-SWE-agent. These are called "harnesses": the machinery that handles instructions to the AI model, the tools available to it, the passing of execution results back, and the management of conversation history. The study includes Claude Code as a harness to compare, but it does not examine whether the Claude models themselves are better or worse than others.
The non-peer-reviewed preprint "What Does a Harness Buy? Tokens, Mostly." asks how much benefit spending on a harness's extra features actually buys. However, failing to find a statistically clear difference in success rates is not the same as confirming that performance is equivalent within a given tolerance. To interpret the findings, you need to look not only at success rates but also at the cost per success and at which failures a harness's features can prevent.
How Do Performance Differences Look Next to Rerun Variability?
The researchers used SWE-bench Verified, a benchmark that measures the ability to fix real software bugs. Of the 500 tasks, they excluded 8 that require an external web service for verification, and split the remaining 492 into 45 hard tasks and 447 others. The 45 hard tasks consist of 42 with an estimated working time of 1 to 4 hours and 3 that exceed 4 hours. In every trial, the number of model calls was capped at 300.
On the 45 hard tasks, five models were compared: Qwen3.6-35B-A3B, Qwen3.8-27B, DeepSeek-V4-Flash (0731), GLM-5.3-Flash, and HY4-Preview. On the remaining 447 tasks, two Qwen models were used. For reference, Claude Opus 5 was also run, but only on Claude Code, so it is not part of the three-harness comparison.
When the harness was changed for the same model on the 45 hard tasks, the median share of tasks with inconsistent outcomes was 13%. Rerunning the same model–harness combination also gave 13%, while changing the model itself gave 22%. In other words, the variation in results from swapping harnesses was about the same as the natural variability from repeating an identical configuration. Note, though, that this 13% does not mean the success rate moved by 13 percentage points. If one task flips from success to failure and another from failure to success, the total number of successes can stay nearly the same while per-task disagreement rises.
The difference in success counts between harnesses was 2 to 5, versus a maximum of 3 between reruns of the same configuration. However, the rerun comparison covers only 10 pairs, and not every configuration was repeated. So the absence of a clear performance difference in a 45-task test does not prove that the harnesses perform identically.
Using more tasks allows performance differences to be assessed in more detail. Comparing Claude Code and mini-SWE-agent on the 447-task set with Qwen3.6, among 445 tasks where both runs were valid, successes were 310 and 318 respectively. The success-rate difference (Claude Code minus mini-SWE-agent) was −1.8 percentage points, with a 95% confidence interval of −5.2 to +1.6 points. With Qwen3.8, across 444 shared valid tasks, the counts were 347 versus 353, a difference of −1.4 points with a 95% confidence interval of −4.2 to +1.5 points.
The team did not merely test for significant differences; it also tested statistical equivalence. For both models, the 90% confidence interval fell within the pre-specified ±5-point margin, so by this criterion Claude Code and mini-SWE-agent were judged equivalent. This does not guarantee that the difference is zero, however. Also, when Qwen3.8 was run on OpenCode, a statistically significant performance gap remained even after correction for multiple comparisons, so the three harnesses were not all equivalent.
The number of tasks used in a test also matters a great deal. The paper estimates that, assuming a 14% outcome-disagreement rate, a 45-task test has only a 50% chance of detecting a 12.9-point performance difference. With 447 tasks, a 5.2-point difference could be detected with 80% power. Confirming small improvements requires testing on that many more tasks.
Clear Differences Emerged in Cost per Success
While differences in success rates were not necessarily clear, large differences appeared in running costs. Section 4.4 and Table 2 of the paper compare the cost of the first attempt on the same 45 hard tasks using GLM-5.3-Flash. Costs are computed from API pricing at the time of the study, separately for cached input, uncached input, and output tokens, and are shown in Chinese yuan (CNY). The table below includes the paper's figures along with this article's own estimate, derived by dividing average per-attempt cost by the number of successes.
| Harness | Avg. cost per attempt | Successes | Cost per success (this article's estimate) |
|---|---|---|---|
| Claude Code | 1.81 yuan | 22/45 | ~3.70 yuan |
| mini-SWE-agent | 0.97 yuan | 20/45 | ~2.18 yuan |
| OpenCode | 0.56 yuan | 18/45 | ~1.40 yuan |
Cost per success was calculated by dividing total cost, including failed attempts, by the number of tasks actually solved. Specifically, the formula is "average cost per attempt × 45 ÷ successes": Claude Code is 1.81 × 45 ÷ 22 = 3.70227… yuan, mini-SWE-agent is 0.97 × 45 ÷ 20 = 2.1825 yuan, and OpenCode is 0.56 × 45 ÷ 18 = 1.40 yuan. All use the rounded average costs printed in the paper, rounded to two decimal places. By this calculation, Claude Code's cost per success is about 2.6 times OpenCode's, showing a large cost gap even after accounting for success rates.
However, the difference in success rates on the 45 hard tasks is not statistically clear, so it cannot be concluded that more expensive harnesses succeed more often. Nor does this figure represent the expected cost of retrying until success or the cost incurred in real-world work. Success rates alone cannot capture the quality of the fixes or the time required, and actual adoption costs also involve factors such as product subscription fees. This estimate, based on API pricing, must also be distinguished from the cost of running Qwen in a local environment.
So why does cost vary so much when the same AI model is used? The research team examined GLM-5.3-Flash's first input and subtracted the 554 tokens corresponding to the task description to estimate each harness's fixed input portion. The results were 16,581 tokens for Claude Code, 7,025 for OpenCode, and 829 for mini-SWE-agent. This is not a direct count of the system prompt alone, but an estimate derived from the residual, which also includes tool definitions and message formatting. The length of tool outputs, by contrast, averaged about 1,100 to 1,200 characters per call, with no major difference among the three.
The harnesses also differ in design. Claude Code and OpenCode have dedicated tools and conversation-history compression, whereas mini-SWE-agent operates only through bash and does no history compression. In configurations with a large fixed input portion, that token cost accumulates with every model call. The average number of execution steps with GLM-5.3-Flash also differed: 88.1 for Claude Code, 80.2 for mini-SWE-agent, and 55.1 for OpenCode.
That said, a smaller fixed input portion does not always mean lower total cost. With DeepSeek, mini-SWE-agent averaged 141 steps versus 104 for Claude Code, the reverse of the relationship seen with GLM. In addition, cache pricing applied to 96 to 99% of input tokens, but with the price table used in the study, the cost ranking of the harnesses did not change even after accounting for cache discounts.
To judge whether an extra feature is worth its cost, you need to look beyond token consumption and check what kinds of failures that feature actually prevents. The study provides a concrete example from the Qwen3.8 runs.
Does Preventing Truncated Output Affect Success Rates?
With Qwen3.8 on the 447 tasks, OpenCode's success rate was 7.9 points lower than Claude Code's, with a 95% confidence interval of 4.3 to 11.5 points. Compared with mini-SWE-agent, it was 9.2 points lower, with a 95% confidence interval of 5.5 to 12.9 points. Even after Holm correction for the six comparisons, OpenCode's performance gaps with the other two harnesses remained statistically significant.
Examining the causes of failure, 33 OpenCode attempts ended because they hit the model's output limit, versus just 1 for Claude Code. Claude Code has a mechanism that prompts the model to continue when its output is cut off partway. In fact, when the analysis was restricted to tasks that did not hit the output limit, the success-rate gap between the two roughly halved. However, this analysis alone cannot establish that the presence or absence of the continuation feature caused the performance gap.
The team also ran an experiment porting the continuation prompt into another configuration. Of the 41 tasks involved, results improved on 6, worsened on 2, and were unchanged on 33. No statistically reliable improvement was found in this comparison, so it could not be said that simply adding a continuation prompt closes the gap with Claude Code.
The test conditions were also not fully uniform. For Qwen run in a local environment, Claude Code was allowed up to 36,000 seconds of execution time, whereas OpenCode stopped at its own 5,760-second limit. The authors acknowledge this time-limit difference as an uncontrolled factor. The software versions compared were 2.1.259 to 2.1.273 for Claude Code, 1.18.30 to 1.18.31 for OpenCode, and 2.4.6 for mini-SWE-agent, and the same differences may not persist in future versions.
The key takeaway is not the number of features a harness has, but the perspective of evaluating specifically which failures each feature can prevent. By classifying failures that actually occur, such as interruptions caused by output limits, and measuring how much an added feature resolves them, it becomes easier to judge whether the spending on that feature is justified.
When Does Harness Design Pay Off?
On the other hand, depending on its design, a harness may produce large differences in an AI's problem-solving ability. A separate non-peer-reviewed preprint, "An Empirical Study of Harness Design for Coding Agents," published by Run-Ze Fan and colleagues on September 17, 2026, fixed the basic execution loop of the AI agent and varied how planning is done, the operating interface, and how conversation history is managed, then compared the effects.
The study covered four models: Nemotron-3 at 30B, 120B, and 550B, and Mistral-Medium-3.5-128B. It evaluated 176 configurations using 500 SWE-Bench Verified tasks and 89 Terminal-Bench 2.1 tasks. When the upper limit on the amount of information a model can handle at once was varied across 32k, 64k, 96k, and 128k tokens, history management had a larger effect the smaller the limit was. For models weak at operating through bash, introducing dedicated tools tended to help with problem solving.
However, the effect of a feature varies by model and task, and does not always work in one direction. For example, in a Mistral comparison with the limit set to 128k tokens, planning enabled, and the history-management method unified, the configuration with dedicated tools had a SWE-Bench success rate 23.2 points higher than the bash-only configuration. On Terminal-Bench, however, the bash-only configuration was ahead by 6.7 points. Because this comparison also changed how instructions are given, file-state tracking, and post-change diagnostics at the same time, the performance gap cannot be interpreted as stemming only from the number of dedicated tools.
Furthermore, this study ran each task only once per configuration and did not cover every combination of design elements. Its models and tests also differ from those in Liu et al.'s study, so it is not appropriate to treat it as an independent replication or refutation. Rather, it is best seen as research examining the conditions under which harness design takes effect, from a different angle. The results of the two studies alone cannot be used to judge the overall merits of harnesses including top-tier closed models, or performance differences across real-world work in general.
Whether third parties can reproduce the results also deserves attention. Liu et al.'s public GitHub repository contains a 50-column data table covering 8,032 trials, along with analysis scripts. However, 8,032 is not the number of independent tasks but the total number of trials run under multiple conditions. Costs and other figures can be recalculated from the public data, but the full execution logs and the experimental environment, such as Docker images, are not included. Even if the table's aggregate results can be reproduced, it would not amount to independently replicating the actual behavior of the AI agents.
The study also restricted external network access. In a preliminary test that allowed connections, 71% of the DeepSeek and Qwen trials fetched fixes from GitHub, so the team discarded those trials and, in the formal tests, blocked all communication other than to the models' endpoints. This was a measure to prevent existing fixes from being retrieved externally during problem solving, so that the capabilities of the model and harness could be properly evaluated. It does not indicate that the problems leaked into training data or that the models cheated intentionally.
Together, the two studies suggest that the number of features or a simple success rate is not enough when evaluating a coding-AI harness. To compare harnesses in a real development environment, it is necessary to standardize the model and a representative set of tasks and to check the variability of results when the same configuration is repeated. It is then important to measure the number of successes and total token cost, and to analyze causes of failure such as truncated output or insufficient history. Matching the failures that an extra feature can prevent against the added cost it brings is the key to judging what a harness is truly worth.