On October 9, 2026, AI evaluation company Vals AI announced that running multiple AI agents as a team raised the cost of developing a web app to 1.8 to 5.1 times that of a single-agent setup. A statistically significant quality improvement was confirmed in only one of the four conditions. The case study used GPT-6 Sol and Claude Opus 5.5 and compared performance across 50 web app development tasks.
However, team setups did shorten development time under some conditions, so the value of the extra cost cannot be judged on quality alone. When assigning development work to multiple AIs, the effect has to be evaluated according to whether the goal is higher-quality output or faster development.
50 apps built, with UI test pass rates compared
The experiment used Vibe Code Bench, a benchmark in which a complete web app is built from a product specification, and compared each model in a single-agent configuration and a team configuration.
The amount of compute allocated to reasoning was set at two levels, "medium" and "maximum." Combining two models, two reasoning settings, and two configurations produced eight conditions in total. In each condition, the same 50 app development tasks were run once each.
In the team configuration, a lead agent could assign work in parallel to up to five subagents. Subagents were not allowed to delegate work further to other agents.
The finished apps were evaluated by actually operating them in a browser. The score is each app's UI test pass rate, averaged across the 50 apps.
Under the evaluation method published by Vals AI, a test is judged as passed when at least 90% of the operations that make up the test succeed.
This means a score of 90% does not imply that "45 of the 50 apps were fully completed." Nor is it a measure that guarantees overall software quality, such as maintainability or security.
The comparison results that Vals AI published on its official blog are as follows.
| Model / reasoning setting | Configuration | UI score | Average cost per app | Median run time |
|---|---|---|---|---|
| GPT-6 Sol, medium | Single | 77.6% | $1.22 | 12.4 min |
| GPT-6 Sol, medium | Team | 84.9% | $3.07 | 11.9 min |
| GPT-6 Sol, maximum | Single | 89.0% | $4.82 | 38.4 min |
| GPT-6 Sol, maximum | Team | 90.4% | $8.54 | 27.6 min |
| Claude Opus 5.5, medium | Single | 91.5% | $4.08 | 20.3 min |
| Claude Opus 5.5, medium | Team | 91.2% | $9.10 | 26.6 min |
| Claude Opus 5.5, maximum | Single | 89.8% | $23.77 | 74.7 min |
| Claude Opus 5.5, maximum | Team | about $122 | about $122 | 174.1 min |
Note: All conditions were evaluated on the same 50 app development tasks. Cost is an average and run time is a median, so the aggregation methods differ. The conditions used to calculate cost are described below.
The only case in which team setup produced a statistically significant improvement was GPT-6 Sol at medium reasoning. The UI score rose 7.3 points, from 77.6% for the single agent to 84.9% for the team.
The p-value from a paired t-test was 0.005, and the 95% confidence interval for the difference was +2.3 to +12.4 points.
With Sol at maximum, by contrast, the improvement was only 1.4 points. With Opus 5.5, the score fell 0.3 points at medium and rose 3.4 points at maximum, but neither difference was statistically significant.
That said, the absence of a significant difference does not prove that team setups have no effect.
For example, the 95% confidence interval for the score difference for Opus at maximum is wide, from -1.4 to +8.3 points. This also shows that the experiment alone cannot determine the size of the improvement from team setups with sufficient precision.
Costs were driven less by code generation than by huge volumes of input tokens
According to Vals AI, most of the added cost from using teams came from processing cached input tokens.
Each subagent holds the product specification, work progress, and other information as its own context. Because this information has to be fed in again each time the model is called, the total volume of input tokens rises sharply when multiple agents work in parallel.
Prompt caching can reduce the cost of processing input, but tokens reused from the cache are not free.
The most striking case was Claude Opus 5.5 at maximum reasoning.
The cached input processed by the group of subagents reached a median of 224 million tokens per app. That is very large compared with 55 million tokens for the single-agent configuration and 17 million tokens for the lead agent in the team configuration.
These figures, however, do not represent newly generated code or text. They are cumulative totals of input tokens the model read repeatedly.
Note also that simply adding up the median of each group does not give the median for the whole team.
Interestingly, the models worked differently within their teams.
The GPT-6 Sol lead tended to divide up work early in development and then integrate and test the finished parts itself.
In nine apps whose scores improved by more than 20 points at medium reasoning, the team configuration added features that the single-agent configuration had not implemented. In some cases, the lead also found problems such as mismatches between input fields on a screen and the database definition.
In other words, dividing the work and checking during integration appears to have reduced missing features and inconsistencies.
Claude Opus 5.5, meanwhile, tended to document the interfaces and specifications needed between features first, and then delegate work in stages: foundation implementation, feature additions, and testing.
With Opus at maximum, the median time to first deployment was 51 minutes for the single-agent configuration and 56.8 minutes for the team, not a large difference.
However, total development time was 74.7 minutes for the single agent and reached 174.1 minutes for the team.
In other words, even though the time needed to get the app running for the first time was nearly the same, the steps after that took far longer in the team configuration.
There was also a large difference in the work time spent on post-deployment testing.
The median of total work time summed across agents was 9.6 agent-minutes... correction: 9.6 agents' worth of time for the single configuration, versus 67.8 agents' worth for the team configuration.
This is the sum of the time spent by multiple agents and differs from the time a user actually waits.
It is certain that the team configuration put far more work into testing, but it cannot be said that this led to a statistically significant improvement in UI scores.
The value of faster development is separate from the value of higher quality
In the experiment, there were conditions in which team setups shortened development time.
The most notable was GPT-6 Sol at maximum reasoning.
Median run time was 38.4 minutes for the single-agent configuration and 27.6 minutes for the team, a reduction of about 28%.
This was calculated from the figures Vals AI published as (38.4 − 27.6) ÷ 38.4 × 100. It compares the median run time of each configuration and is not an average of the reduction for each of the 50 apps.
Because each condition was run only once, the same percentage of time savings will not necessarily hold for other development tasks.
Under this condition, the average cost per app rose from $4.82 to $8.54. The UI score rose only from 89.0% to 90.4%, and no statistically significant improvement was confirmed.
Even so, when finishing development quickly has value, paying extra for a team setup can make sense.
For example, if it reduces the time developers wait before starting their next task, the value created could exceed the increase in cost.
Conversely, if the only goal is to raise the quality of the finished product, these results do not provide sufficient grounds for spending extra on a team setup.
It is also not appropriate to divide the increase in average cost by the reduction in median run time to calculate a "cost per minute saved." That would combine two different statistics, a mean and a median, and would not reflect the relationship between cost and time for individual apps.
Besides forming a team, another option is to increase the reasoning allocated to a single agent.
For the GPT-6 Sol single-agent configuration, raising the reasoning setting from medium to maximum improved the UI score by 11.4 points, from 77.6% to 89.0%. That difference was statistically significant.
However, comparing the single-agent configuration at maximum with the team configuration at medium, the former scored 4.1 points higher, but the difference was not statistically significant.
Meanwhile, the single-agent configuration at maximum cost about 1.6 times as much and took about three times as long.
Increasing reasoning, therefore, is not a universal alternative to forming a team either.
What matters is that the most efficient approach changes with the nature of the development task.
In a technical write-up on its multi-agent system published in June 2025, Anthropic cited research work, where information can be investigated independently from different directions, as the kind of task in which parallel processing by multiple agents tends to be effective.
It also explained that coding offers fewer fully independent tasks than research does.
For example, even if separate agents implement a web app's screens and its database, fixes will be needed at integration if data formats and feature specifications do not match.
Being able to run multiple agents at once does not necessarily shorten the overall working time.
Anthropic's article also cites a figure that multi-agent systems use about 15 times as many tokens as ordinary chats.
That figure, however, compares ordinary chats with multi-agent systems. The 1.8 to 5.1 times shown by Vals AI is the cost ratio between single-agent and team configurations.
Because both the comparison targets and the metrics measured differ, the two cannot be compared directly.
Actual development costs and performance depend on your environment
Care is needed in applying these findings directly to your own development environment.
First, the way cost was calculated differs by model.
For GPT-6 Sol, Vals AI assumed that the reported token usage included subagent usage and applied base rates of $2 per million input tokens, $0.20 for cached input, and $10 for output.
However, surcharges apply to long inputs exceeding 272,000 tokens. Because token usage was reported as a total for the entire run, the researchers say they could not accurately calculate surcharges for individual requests.
For Claude Opus 5.5, by contrast, the list-price-based cost reported by Claude Code was used.
In the team configuration at maximum reasoning, one run timed out, but its cost was estimated at about $189 from token usage and included in the average calculation.
In other words, the amounts in the table were calculated under conditions set by the research team, and the same bill will not necessarily result in every environment.
Furthermore, the experiment did not vary only the number of agents.
The team configuration was given additional instructions encouraging division and integration of work. For Opus, the 10-minute run limit that had applied to subagents running in the background was also lifted.
The execution platforms also differed: the Agents API was used for GPT-6 Sol and Claude Code for Claude Opus 5.5.
The differences in execution environment may therefore have affected the performance gaps between models. The observed ways of dividing work alone cannot establish the cause of the differences in cost or quality.
These results are a case study published on the official blog by Katrina Drozdov of Vals AI; the comparison of team configurations was not itself published as a peer-reviewed paper.
Because each configuration built the 50 apps once each, the stability of results when the same tasks are run repeatedly has not been confirmed.
The reported confidence intervals and p-values are based on the variation in score differences observed across the 50 apps. Because AI agent runs involve stochastic variation, running the same tasks several times will not necessarily yield the same results.
For real-world adoption, a realistic approach would be to first prepare the same tasks and completion criteria and compare the method of increasing a single agent's reasoning with the method of dividing work among multiple agents.
In doing so, it is necessary to record actual billed amounts and time to completion separately, in addition to test pass rates. It is also important to run the same tasks repeatedly and check whether differences in quality, cost, and run time reproduce consistently.
What this research shows is the reality that adding more AI agents does not necessarily improve development quality or shorten development time.
There are certainly situations where multiple agents are effective. But whether the benefits of dividing work outweigh the added costs of sharing information, integration, and testing depends on the nature of the work.
As AI-driven software development spreads, it will become more important to judge which tasks to divide up and where to put additional compute, rather than simply increasing the number of agents.