When companies entrust AI agents with coding work, the number they see before signing a contract is the vendor's listed price—"$X per 1,000 tokens." Yet Uber, after pushing AI adoption company-wide in 2026, burned through its entire annual budget in just four months, forcing it to impose usage caps in June. If choosing a model with a lower per-token price is supposed to save money, why do real-world costs end up exceeding estimates?
An internal benchmark Databricks published on July 9 points to the answer. Anthropic's Sonnet 5 is roughly 1.7x cheaper per token than Opus 4.8, yet the cost per task came out to $2.09—higher than Opus 4.8's $1.94. Conversely, GLM 5.2 delivered quality on par with Opus 4.8 while costing $1.28 per task, 34% cheaper. Per-token pricing simply doesn't predict the actual bill.
Sonnet 5's miscalculation: surpassing Opus when measured per task
This benchmark was published by Databricks' engineering team on its official blog, and it evaluates real tasks that engineers handle daily against the company's own codebase spanning millions of lines. Co-authors include Databricks CTO Matei Zaharia (who also holds a position as Associate Professor of Computer Science at UC Berkeley). The results split clearly into three distinct capability tiers depending on the model-harness combination, with GLM 5.2 in the top tier delivering quality statistically on par with Opus 4.8 while costing $1.28 per task—34% less than Opus 4.8's $1.94. The blog also notes that models from OpenAI, Anthropic, and open source each occupy different points on the optimal cost-quality trade-off (the Pareto frontier), meaning a procurement policy that relies on a single vendor's products cannot reproduce the performance level measured here.
The problem is Sonnet 5. Databricks' blog reports that despite being roughly 1.7x cheaper per token than Opus 4.8, Sonnet 5 ended up costing $2.09 per real task versus Opus 4.8's $1.94, and its task completion rate was 6 points lower at 81% (compared to Opus 4.8's 87%). The blog attributes this primarily to the fact that Sonnet 5 ran longer and read more content before completing tasks, resulting in token consumption 1.9x that of Opus 4.8. In other words, the ranking based on per-token pricing that informed the initial estimate didn't match the ranking based on actual billed amounts.
How the gap in token consumption undermines the appearance of cheapness
Databricks summarizes the reason for this reversal between per-token price and per-task cost as follows: "variance in inference efficiency across models makes per-token pricing a poor proxy for actual task cost." To answer the same question, one model might read only the essentials briefly and produce an answer, while another might re-read code repeatedly and think at length before answering. Sonnet 5 ran longer and read more than Opus 4.8, ballooning its token consumption to 1.9x—and that increase in consumption outweighed the advantage of its lower per-token price.
Databricks' blog doesn't specify which of Anthropic's pricing tiers this "roughly 1.7x cheaper" comparison is based on. Separate from its standard pricing (Sonnet 5 at $3 per million input tokens and $15 output; Opus 4.8 at $5 and $25), Anthropic offers introductory pricing valid through August 31, 2026 (Sonnet 5 at $2 input and $10 output), which works out to roughly 2.5x cheaper than Opus 4.8 per token during that window. Since the 1.7x ratio Databricks cites matches the standard-pricing ratio (5/3, 25/15), it appears to have been calculated using standard pricing. Recalculating with the same token consumption but using introductory pricing instead, Sonnet 5's per-task cost drops to roughly $1.39—two-thirds of $2.09—which would actually undercut Opus 4.8's $1.94. This is a caveat worth keeping in mind: the conclusion itself can flip depending on the pricing assumptions used.
In Sonnet 5's case, at least under standard pricing, its token consumption piled up to a degree that its lower per-token price couldn't offset, and combined with its inferior completion rate, it landed at a higher per-task cost than Opus 4.8. A similar phenomenon was also confirmed in the March 2026 academic paper "The Price Reversal Phenomenon."
Lingjiao Chen and colleagues (with Zaharia also listed as a co-author) evaluated eight frontier reasoning models across 12 types of tasks and found that in about 32% of model comparisons, the model with the lower list price ended up costing more in practice. As one example, they report that Gemini 3 Flash, despite listing 80% cheaper than GPT-5.4, ended up 38% more expensive in actual cost across the full task set. The same paper found that price reversals reached as much as 28x in magnitude, and that even for the same question, thinking-token consumption varied by up to 900% between models. The primary driver of these price reversals is the large variance across models in how many thinking tokens they consume per query and how many rounds of environment interaction they require—retry costs following failures are not the direct cause. Decisions to switch to a cheaper model based on per-token price alone tend to overlook exactly this variable: consumption volume.
The complexity of tasks an agent can handle in a single session keeps growing year over year. According to measurements by AI evaluation organization METR, the length of tasks (measured in human-expert-equivalent time) that AI agents can complete at a given confidence level has been doubling roughly every seven months from 2019 through 2025. As tasks grow longer and more complex, the sheer volume of tokens consumed per attempt increases, meaning that differences in consumption between models will have an even larger impact on actual costs going forward.
Per-task cost in yen: the real gap between the three models
Converting at ¥162 to the dollar (an approximate rate as of mid-July 2026), the per-task costs Databricks measured for the three models work out as follows. This is a simple conversion applying the exchange rate to the dollar figures and doesn't account for currency fluctuations or individual contract discounts. Even so, putting the numbers on a common scale helps development teams budgeting in Japanese yen grasp the relevant magnitude.
- GLM 5.2: $1.28 → approx. ¥207/task
- Opus 4.8: $1.94 → approx. ¥314/task
- Sonnet 5: $2.09 → approx. ¥339/task
The gap between the cheapest option, GLM 5.2, and the most expensive, Sonnet 5, is only about ¥132 per task. But for a development team running hundreds of tasks a day and thousands per month, that gap compounds into a difference of an entirely different order of magnitude. What gets compared at contract time is the vendor's listed per-token price, but the actual per-task cost is determined by two empirical measurements—token consumption and completion rate—meaning a gap between the contract-stage estimate and the post-deployment actual cost is built in from the start.
For example, consider a hypothetical development team of 20 engineers handing 50 tasks a day each to AI agents—1,000 tasks processed per day. Applying the per-task cost gap between Opus 4.8 and Sonnet 5 (roughly ¥25 per task) to this scale works out to about ¥25,000 per day, or roughly ¥500,000 per month over 20 business days (this is not a figure from any real, specific company—it's purely a hypothetical scenario meant to illustrate scale). A seemingly small difference in per-task cost balloons into a sum that's hard to ignore once team size grows.
How Uber's $1,500/month cap connects to per-task cost
After Uber pushed company-wide adoption of AI agents in 2026—even setting up an internal leaderboard—it burned through its entire annual AI budget in just four months. According to reporting by TechCrunch and Bloomberg, Uber set a usage cap in June 2026 for agentic coding tools such as Claude Code and Cursor: $1,500 per employee per month, per tool (not a combined cap across all tools). The scale behind this budget overrun includes roughly 95% of engineers using such tools and AI agents generating about 10% of the company's code.
Applying this per-tool cap to the per-task costs Databricks measured reveals how the number of tasks that can be completed varies significantly depending on the model chosen. If we simply divide the $1,500-per-tool monthly allowance by per-task cost alone, it works out to roughly 1,172 tasks for something equivalent to GLM 5.2, about 773 tasks for Opus 4.8, and about 718 tasks for Sonnet 5 (Uber's actual usage models and task composition haven't been disclosed publicly—this is purely an estimate applying Databricks' figures to the cap amount). Even with the same $1,500 allowance, the number of tasks that can be completed ranges from 718 to 1,172 depending on the model chosen—a gap of up to 39%. While Uber's budget overrun was primarily driven by a rapid surge in usage volume, it does serve as circumstantial evidence that a procurement approach based solely on per-token pricing—one that omits completion rate as a variable—may have been at play.
When leadership estimates AI budgets using "per-token price × expected usage volume," it's hard to account for the risk that the amount of work actually completed will fall short of expectations depending on real-world completion rates. In Uber's case, no specific model's completion rate has been named as the direct cause of the budget overrun. But in a situation where 95% of engineers routinely use agentic tools and AI generates 10% of the company's code at that scale, the underlying structural risk—that procurement based solely on per-token price misjudges actual consumption—applies broadly.
Cheaper models aren't automatically a bargain: harness choice can swing cost by 3x
Another notable finding from Databricks' testing is that even with the same model and the same reasoning-effort setting, cost varied significantly depending on the execution environment (harness). Comparing the same model across vendor-provided harnesses like Claude Code and Codex versus Databricks' own lightweight in-house harness, "Pi," they found cases where cost per task differed by more than 2x with no change in quality. The main driver of this gap was how much context each harness sends to the model per exchange. Pi sent roughly one-third less context per exchange than the other harnesses, staying tightly scoped to the task at hand and completing tasks in fewer round trips.
A procurement decision that focuses solely on model selection while leaving the harness at its default setting is only looking at half of the cost optimization picture. Building on this insight, Databricks says it has invested in "Omnigent," an internal platform that lets teams flexibly switch between model-harness combinations. The intuition that "switching to a cheaper model saves money" tends to break down the moment it's reduced to a question about the model alone.
The limits of a single-codebase evaluation, and the next step toward measuring TCO
Databricks built this evaluation in-house for two reasons. First, public benchmarks like SWE-Bench and TerminalBench have their solutions leak into training data over time since the problems themselves are public. Second, they aren't representative of Databricks' own multilingual codebase, which spans more than 10 languages including Scala, Go, Rust, Java, and Python. Because the benchmark was built from real, merged pull requests, Databricks says it enables optimization decisions that don't get in the way of its own engineers. GLM 5.2's 34%-cheaper result, too, is based on this company-specific codebase and task distribution, and there's no guarantee it would reproduce identically at companies with different industries or codebase characteristics.
Zaharia is also a co-author of the March 2026 academic paper "The Price Reversal Phenomenon." The fact that the same reversal phenomenon has now been confirmed both in an internal benchmark and in academic research involving the same individual lends credibility to the idea that this isn't a one-off measurement error. At the same time, since Zaharia is a co-author on both analyses, these results should be understood in the context of being "Databricks' internal evaluation" rather than an independent third-party audit.
Taken at face value, these measurements suggest there are multiple metrics worth comparing when procuring AI coding agents—per-token price is just one of them. Comparing prices without factoring in task completion rate leaves actual token consumption and quality-related risk out of the estimate entirely.
Average token consumption per task isn't determined by the model alone either—it can swing roughly 3x depending on harness design. And above all, what's indispensable is the verification process itself: confirming whether these evaluation results actually reproduce on your own codebase and workload.
Meanwhile, an analysis by Epoch AI shows that when tracking the price of the cheapest model meeting a given performance bar across multiple benchmarks, that price falls at a median annual rate of roughly 50x. This figure reflects the decline in price for the cheapest model meeting a performance threshold—it is not a figure representing the average per-token price across the entire market. It doesn't necessarily mean the real cost of individual tasks falls at the same pace, which points in the same direction as Databricks' reversal phenomenon. Databricks itself says that, informed by this analysis, it has made the call to route routine, templated work—like flag toggles or configuration changes—to cheaper models in the Haiku or GPT 5.4 Mini class.
A vendor's list of per-token prices is merely a starting point. Whether an enterprise can go the distance—remeasuring completion rate, token consumption, and harness dependency against its own codebase and task distribution, then actually feeding those results back into procurement decisions—is what will separate effective AI budget management from the rest.