Anyone who has stared at a benchmark comparison table trying to decide between GPT-family and Claude-family models knows the feeling: you agonize over a few points' difference in scores, only to find that once the model is actually wired into a real workflow, it doesn't deliver the accuracy you expected. Have you ever sensed that disconnect?
A verification result NVIDIA published on August 21, 2026 shakes that very premise. In ARC Prize's own measurement, Anthropic's Claude Opus 5, even with high-reasoning settings enabled, scored only about 30% on its own. Yet when run on NVIDIA's in-house general-purpose agent framework, "AVO," the same model reportedly cleared every one of the 183 public ARC-AGI-3 levels, posting a perfect RHAE score of 100.00. NVIDIA itself cautions that the two measurements used different evaluation frameworks and aren't a direct comparison — but the gap illustrates just how much a model's ultimate capability can shift depending on the design that surrounds it, even when the underlying model is identical.
How the same model went from "30 points" to a perfect score
The numbers NVIDIA laid out in its official technical blog are specific. Running Claude Opus 5 on "AVO" (Agentic Variation Operators), a general-purpose agent framework, the model completed all 183 public ARC-AGI-3 levels, achieving a perfect score of 100.00. This metric is ARC Prize's RHAE (Relative Human Action Efficiency) — a score that only reaches 100 when every level is completed and the agent's action efficiency matches or exceeds the human baseline for first-attempt performance. The run required 6,624 environment actions in total. By contrast, when Claude Opus 5 was measured using ARC Prize's own general-purpose harness with high-reasoning settings, its score topped out around 30%.
However, as NVIDIA itself explicitly notes, the two measurements differ in agent backend, observation representation, memory management, and evaluation framework — this is not "a controlled ablation comparison," nor does it "directly measure AVO's standalone performance contribution." These are not numbers that can simply be read as a straightforward multiplier. It's best understood as one example showing how far the outcome can shift depending on the surrounding design.
ARC-AGI-3 is a benchmark made up of 2D puzzles that must be solved by inferring the rules on the fly, without prior instructions or explanations — it's used as a gauge of general reasoning ability. AVO itself traces back to research published on arXiv in March 2026, originally aimed at automatically optimizing CUDA kernels. That earlier paper reported that seven days of autonomous execution achieved performance gains of up to 10.5% over FlashAttention-4 and up to 3.5% over cuDNN. What's distinctive this time is that NVIDIA repurposed that same architecture for an entirely different task domain — ARC-AGI-3 — demonstrating its generality.
Adel El Hallak, VP of Product at NVIDIA's AI division, explained the thinking behind the result this way: "People tend to interpret an agent as just a model's API, but in reality it's the model itself plus the scaffolding around it — the tools it uses, the runtime, and the skills and libraries it can access." AVO's architecture includes a built-in "supervisor" agent that, in El Hallak's words, "nudges the main agent like a CEO would" whenever the main agent veers off course, starts exploring dead ends, or begins re-traversing a path it has already tried.
NVIDIA didn't use one of its own models for this test — it chose Anthropic's Claude Opus 5. The two companies do overlap in the model layer, but this isn't a simple competitive relationship: NVIDIA has invested up to $10 billion in Anthropic and formally announced a technology partnership to co-optimize models and GPU architecture. NVIDIA doesn't have a frontier-class closed model of its own; the models it publishes, such as Nemotron, remain open-source. Given that position, having a partner company's latest model achieve a perfect score once wrapped in NVIDIA's harness lends more credibility to the claim that AVO's benefits are model-agnostic. Put another way, the very choice of model reinforces NVIDIA's argument that the value of the harness doesn't depend on which model sits underneath it.
What exactly is a "harness," and why can two settings triple a score?
The term "harness" here refers to the entire surrounding design that actually enables a model to carry out a task: how inputs and outputs are passed back and forth, how state is retained, how tool calls are controlled, and how the system recovers from failures. It's the design of the "undercarriage" — what gets fed to the model and how its responses get handled. In a separate blog post dated July 27, 2026, NVIDIA laid out six core principles for harness design: typed inputs/outputs, pass-by-reference, treating actions as code, programmable loop design, explicit object state, and harness APIs the model itself can call.
Backing up the impact of this approach is a result OpenAI published on July 29, 2026. When measured on the public ARC-AGI-3 set using ARC Prize's official harness, GPT-5.6 Sol scored only 13.3%. But when OpenAI reimplemented the harness on top of its Responses API and added two settings — retained reasoning (preserving the model's internal reasoning) and compaction (compressing history) — the score jumped to 38.3%, roughly triple the original. Output token usage, meanwhile, dropped to about one-sixth.
The official harness had been discarding the model's internal reasoning every time it took an action, and truncating older parts of the conversation history once it exceeded a certain length. Simply rebuilding the harness and changing how reasoning continuity and history were handled moved the score by that much.
Databricks CEO Ali Ghodsi has testified to the cost impact of harness choice in concrete terms: "Even with the same model, choosing a different harness — if you pick the wrong one — can dramatically increase your costs." According to the company's own measurements, a task that cost about $0.74 when routed through a harness called Pi cost about $1.94 when routed through the native harness — a more than 2.6x difference in cost for roughly equivalent quality. Without swapping out the model at all, the way it's wrapped alone can produce a completely different outcome in both accuracy and cost — which is exactly why harness design is drawing rapid attention.
NVIDIA itself has published one example involving an open model. According to LangChain's own blog, when LangChain tuned its "Deep Agents" harness for NVIDIA's Nemotron 3 Ultra, the quality score reached roughly the same level as Claude Opus 4.8 (best-run scores of 0.86 for Nemotron 3 Ultra versus 0.87 for Opus 4.8), while the per-run cost came in at about $4.48 — roughly a tenth of Opus 4.8's approximately $43.48. Since this measurement was conducted by parties who co-developed the setup rather than an independent third party, it should be viewed with some caution. Even so, it's consistent with the broader trend that there's substantial room for harness-side tuning regardless of a model's scale or whether it's open or closed.
From competing on model performance to competing on how models are wrapped
Tracing the timeline helps explain why these demonstrations have piled up in such quick succession. It started in March 2026, when the research underlying AVO first appeared on arXiv. That original paper proposed a general method of replacing the mutation and crossover steps of classical evolutionary search with autonomous agents; optimizing GPU attention kernels was just one of several evaluation targets at the time.
The next turning point came in July. NVIDIA published its blog laying out six harness design principles on July 27, and just two days later, on July 29, OpenAI announced that two configuration changes had tripled its ARC-AGI-3 score. Around the same time, Databricks published benchmark results for its own coding agent, showing measured cost differences of roughly twofold depending on the harness used. Then, on August 21, NVIDIA repurposed AVO for the general-purpose ARC-AGI-3 benchmark and produced the most extreme figure yet: a jump from 30% to 100%. In under five months, five separate announcements piled up — a CUDA-optimization experiment, a set of design principles, a threefold improvement from configuration changes, measured cost data, and finally a perfect score.
What this sequence reveals is a clear shift in the AI industry's competitive axis. The term "harness engineering" itself began gaining traction around February 2026, when OpenAI announced that its Codex agent had built roughly a million lines of internal beta software with zero hand-written code (humans instead focused on designing the environment, giving instructions, and building feedback loops). In under half a year since then, the focus has moved from competing purely on a single model's accuracy to competing on the design of the entire agent system — tool calls, memory management, and orchestration included. NVIDIA's AVO announcement stands out among these as the case with by far the largest improvement.
Who stands to benefit? NVIDIA's calculated angle
Lining up these facts makes it fairly clear who benefits from this rush of demonstrations. The most obvious beneficiary is NVIDIA itself. In March 2026, NVIDIA CEO Jensen Huang said at the Morgan Stanley TMT Conference that agentic AI could consume up to a million times more tokens than a standard single-shot prompt. On March 19, he made a similar point on the All-In podcast, saying something to the effect that he would be deeply concerned if an engineer earning a $500,000 salary weren't using at least $250,000 worth of tokens.
Demonstrating that an agent's real-world performance can leap forward depending on harness design becomes, in effect, an argument for handing agents more tasks. And the more tasks agents handle, the more tokens get consumed proportionally, driving up demand for the GPUs that support all of it. NVIDIA's announcement that AVO achieved a perfect score on ARC-AGI-3 functions simultaneously as a technical demo and as an advertisement reinforcing the need for the very computing resources NVIDIA sells.
Databricks occupies a similar position of benefiting from this dynamic. The company has publicly shared its own head-to-head measurements between the Pi harness and its native harness, turning harness-optimization know-how itself into a point of differentiation. If enterprise customers, instead of agonizing over "which model to use," start bringing their "how to combine things" questions to Databricks, that only raises the value of the company's consulting and platform offerings.
On the other hand, the vendors who have competed on model-only performance are the ones whose position is now being called into question. Even accounting for differences in evaluation conditions, if the same model's numbers can swing this dramatically depending on the harness, then benchmark comparisons between models start to lose much of their meaning. Companies have long competed on standalone score superiority with every new model release, but if that gap can be reversed by harness design alone, the relative value of relying on standalone benchmarks as a basis for model selection declines. As far as I can confirm from NVIDIA's official blog and Anthropic's official communications, the former makes no mention of AVO's commercial timeline, pricing, or delivery format (API versus open source), and the latter has issued no official comment on this result at all. Buried in that silence is a telling picture: the yardstick for evaluating a company's own model is increasingly being rewritten somewhere beyond the reach of the company that built it.
Even harnesses have limits — read the numbers with caution
In April 2026, Microsoft researcher Philippe Laban and colleagues published "DELEGATE-52," a benchmark testing 19 LLMs across long-horizon delegated workflows spanning 52 specialized domains. In the main experiment, where models were repeatedly handed document-editing tasks as independent, single-turn sessions, even frontier models including Gemini 3.1 Pro, Claude 4.6 Opus, and GPT-5.4 ultimately corrupted an average of 25% of document content. According to the paper, the cause wasn't forgotten instructions or lost context — it was rare but serious editing errors that quietly damaged documents, accumulating over repeated iterations.
In a separate experiment where the models operated agentically with tools, an additional average degradation of 6 points was observed, and even GPT-5.4, the best-performing model, couldn't escape an additional 3-point decline. These numbers suggest that even careful harness design can't fully prevent this kind of cumulative degradation over long-horizon delegated work.
OpenAI raised its score from 13.3% under ARC Prize's official harness to 38.3% — roughly 2.9 times higher — by reimplementing the harness on the Responses API with retained reasoning and compaction settings, essentially fixing constraints (discarded reasoning and forced history truncation) that existed in the prior harness. NVIDIA's AVO figures, by contrast, place the standalone model's roughly 30% score (under ARC Prize's general-purpose harness with high-reasoning settings) alongside AVO's RHAE score of 100.00 — and as NVIDIA itself acknowledges, this is a comparison between two entirely separate evaluation environments that differ in agent backend, observation representation, and memory management, among other things.
That's precisely why these two "harness improvements" are not the same kind of thing. OpenAI's gain is a delta from reimplementing an existing harness; NVIDIA's AVO result is the endpoint of deploying a purpose-built architecture from scratch. Not only does the scale of design effort invested differ substantially, but the underlying evaluation environments being compared aren't the same either — so even though the resulting multipliers look similar in magnitude, they can't simply be lined up to declare one superior to the other. What both cases do share, though, is the plain fact that evaluation results shifted dramatically without changing the underlying model at all.
What this means for Japanese companies, and where the next turning point lies
This structural shift isn't irrelevant to Japanese companies either. According to an internal survey by GMO Internet Group, the rate of AI agent adoption in business operations jumped from 43% in December 2025 to 71.4% by March 2026, with monthly time savings per employee reaching 53.9 hours — up 7.0 hours from the previous survey. Given this latest demonstration that outcomes for the same model can swing by multiples depending on harness design, domestic companies' AI agent investment decisions may have room to shift focus — from "which model to contract for" toward "how to build the system around it."
Whether the picture NVIDIA is painting actually plays out depends on just how much token consumption AVO can genuinely drive in practice. The first real tests of this will come in how agent-related GPU demand is discussed in NVIDIA's next earnings report, and in whether model vendors like Anthropic move to strengthen their own in-house harnesses. If the ratio Jensen Huang described — agents consuming up to a million times more tokens than a standard single-shot prompt — bears out in real-world deployment, then what drives GPU demand going forward will be agent workload volume itself, far more than the number of new model contracts signed.