On September 4, 2026, Artificial Analysis, a US-UK organization that measures AI model performance, revised its "Intelligence Index," its overall scale for evaluating leading frontier models, to version 4.2. OpenAI's latest frontier model, "GPT-6 Astra," appeared on September 3, 2026. The old v4.1 index scored it level with its predecessor, a result that was questioned because it diverged from measurements by other organizations and from OpenAI's own evaluations. In response, Artificial Analysis pulled forward elements from the next major version, which is still in development. The new index adds real-world knowledge work and large-document reasoning, and Astra rose to a clear second place, four points ahead of its predecessor. Evaluators are worried about existing benchmarks becoming saturated and being overfit. What does that reveal about cutting-edge AI, and what new challenges does it raise?
New Metrics Reveal Astra's 4-Point Lead and Correct the Earlier Tie
The correction began with a discrepancy in the initial measurements of GPT-6 Astra, which was released on September 3, 2026. In an evaluation by the AI research organization Epoch AIAstra ranked first among 267 models across more than 50 benchmarks, earning 169 points. It also posted a large score gain on the abstract-reasoning benchmark ARC-AGI-3, and OpenAI's own published figures showed it clearly outpacing its predecessor, "GPT-5.6 Sol."
Yet in Artificial Analysis's Intelligence Index v4.1, which is widely consulted in practical settings, Astra scored about 51 points, roughly the same as Sol. Because progress on difficult benchmarks was not reflected in the overall index, developers and researchers questioned the index's validity. The concern was that the existing set of measurements had lost its ability to discriminate among the progress of frontier models.
The next day, September 4, Artificial Analysis immediately announced an interim revision to version 4.2. It had been developing "Index v5" for eight months since the v4 release in January 2026, and it split off agent evaluation and document analysis from that work and incorporated them ahead of schedule.
In the revised Intelligence Index v4.2, GPT-6 Astra at maximum reasoning effort (max effort) earned an overall 55 points. That is four points above GPT-5.6 Sol, also at max settings (51 points), placing it alone in second. Anthropic's "Claude Fable 5.1" kept first place overall, followed in third by Meta, with SpaceXAI, Moonshot AI's Kimi, Zhipu AI (Z.AI), and Google behind it.
| Model | Developer | Reasoning effort | v4.2 overall score | Comparison with predecessor |
|---|---|---|---|---|
| Claude Fable 5.1 | Anthropic | Max / Adaptive | Retains first place | Frontier leader |
| GPT-6 Astra | OpenAI | Max | 55 pt | +4 pt vs. Sol |
| GPT-6 Astra | OpenAI | Medium | 52 pt | 1 pt above Sol (Max) |
| GPT-5.6 Sol | OpenAI | Max | 51 pt | Baseline |
The shift in the numbers supports the view that the earlier tie did not reflect stagnating model performance but an evaluation framework with insufficient resolution.
GPQA Saturation and a 40% Private Share Drive a Benchmark Overhaul
The biggest changes in Intelligence Index v4.2 are a thorough replacement of the benchmark lineup and a higher share of private test sets.
The most symbolic step is the removal of "GPQA Diamond," long a leading measure of expert-level scientific reasoning. Frontier models' accuracy had reached "saturation," clustering near the ceiling, so it no longer worked as a yardstick for ranking models. It highlights a structural problem of modern AI benchmarks: even exam questions once considered the hardest lose their discriminating power as models' training advances.
Two demanding new benchmarks were introduced in its place.
The first is "AA-Briefcase" (15% weight), developed in-house by Artificial Analysis. It uses projects designed by industry experts that simulate knowledge work lasting several weeks, requiring models to autonomously complete multiple interlinked tasks drawing on thousands of input source files. Beyond the correctness of verifiable deliverables, the depth of analysis and quality of presentation are scored through rubric evaluation and pairwise judgments. On these tasks Astra gained about 85 Elo points over Sol, reaching a level close to Claude Fable 5.1 and Claude Opus 5.
The second is "GDP.pdf" (10% weight), a long-document reasoning benchmark developed by Surge AI. It measures the ability to synthesize evidence in a single turn across text, charts, footnotes, and exclusion clauses spread over 100 professional PDF documents in 10 domains, totaling 4,592 pages. Its main metric is the "All-pass Rate," under which an answer counts as correct only if it satisfies all 1,275 atomic criteria written by experts. On this extremely strict test, GPT-6 Astra took the top spot with a 33.2% pass rate, ahead of Sol's 28.2% and Claude Fable 5.1's 26.2%.
Another important design change is that the share of private test sets in the overall evaluation data doubled from 20% to 40%. Answer sets for AA-Briefcase, the knowledge evaluation AA-Omniscience, and the physics-reasoning CritPt are protected as private data. When test data is public, it is impossible to rule out benchmark gaming, in which developers intentionally or unintentionally mix it into pre-training or post-training data. Securing private assets is becoming a prerequisite for maintaining evaluation reliability.
The evaluation infrastructure was also made more precise. For the long-context reasoning benchmark AA-LCR v1.1, additional grading prompts were added and ambiguities in the answer data were fixed. For the code-generation benchmark SciCode v1.0.1, the sandbox environment was strengthened to eliminate cases where correct but slow-running code was wrongly disqualified.
Token Efficiency Offsets a 2.5x Price Increase
Alongside the progress in performance measurement, the revision forces a significant change in how the procurement cost of AI models is viewed.
In the API pricing announced by OpenAI, GPT-6 Astra is set at $10 per million input tokens and $50 per million output tokens. Compared with its predecessor GPT-5.6 Sol ($4 for input, $20 for output), the nominal unit price has jumped about 2.5 times. That looks like a major cost increase at first glance, but Artificial Analysis's measured data shows a different picture.
Compared with nearly all competing models in the frontier tier, Astra consumes markedly fewer output tokens to complete the same task. It showed high "output token efficiency," suppressing excessive idle thinking and verbose output and reaching answers in fewer steps.
Thanks to this reduced token consumption, OpenAI continues to hold a place at the front line alongside Anthropic, Meta, and Z.AI on the "Pareto frontier," which plots effective cost per task on the vertical axis against intelligence score on the horizontal axis. Even if the output token price is 2.5 times higher, if the number of tokens required falls to half or less, the final bill can come out equal to or lower than that of the older generation.
Adjustable reasoning effort also provides practical flexibility. GPT-6 Astra set to "medium" reasoning effort scored 52 points on the intelligence index. That exceeds the 51 points of the previous-generation Sol running at maximum effort, while the reduced token consumption keeps the cost per task below Sol's. When companies build these models into their systems, they have entered a stage where cost-effectiveness should be judged by total consumption to complete a task rather than by unit price comparisons alone.
A 33.2% Pass Rate Shows the Long-Document Barrier and a Transition Toward v5
Intelligence Index v4.2 clarified GPT-6 Astra's strengths and position, but it also exposed the limits facing frontier AI.
Even Astra, which took first place on GDP.pdf, has an all-pass rate of only 33.2%. In close reading of professional documents running to 4,592 pages, that works out to errors failing the criteria in two out of three attempts. It has not reached a stage where complete consistency can be guaranteed with a single prompt for the practical materials handled by highly specialized professionals in law, finance, engineering, and similar fields.
The composition of the index itself also retains a transitional character. Although the share of private test sets was raised to 40%, the remaining 60% of items still depend on public or semi-public datasets. As companies' training techniques grow more sophisticated, partial privatization alone cannot completely close off room for gaming.
Artificial Analysis has stated explicitly that v4.2 is "an interim release to keep pace with the rapid evolution of frontier models." The major revision "Index v5," developed over eight months, will be rolled out in stages. The next version is expected to raise the share of private tests further and to introduce in earnest new test cases measuring long-term agent operation and coordination across multiple tools.
What the benchmark revision tells us is that the competition over solving static exam questions is over, and the main battleground has shifted to an overall contest of practicality in handling vast real-world materials and of economics, including output efficiency. For companies and developers, the next things to watch are how far the staged rollout of v5 reduces uncertainty in long-document reasoning, and whether token efficiency materializes as cost savings in their own business workflows.