Artificial Analysis, which evaluates AI model performance, announced its overall benchmark, the Intelligence Index v5, on October 27, 2026. The revision raises the weighting of evaluations that use private test questions, or questions whose correct answers are not public, to 60%. This does not mean that 60% of all questions are private; it refers to the weighting used when calculating the overall score. Whether the questions and answers are private is also a separate matter from whether a model's trained parameters (its weights) are publicly available. Because the tasks being evaluated and the scoring methods have also changed, understanding the score gaps between models requires attention not only to whether a model's weights are open, but also to the intended use and the reasoning settings.
Private evaluations expand to 60%, with new science and code-development tasks
Intelligence Index v5 introduces two new evaluations, "FrontierCode 2.0" and "Terminal-Bench-Science," each accounting for 15% of the overall score. The index is organized into four domains (knowledge work, coding, science, and general capability), each weighted at 25%, and combines nine benchmarks in total. Compared with v4, released in January 2026, v5 has two new evaluations, three updated ones, and four carried over. GDPval-AA v2.1, SciCode, and AA-LCR v1.1, which were previously used, have been dropped.
However, not every newly added science evaluation is private. According to Artificial Analysis's official methodology, the following five evaluations are designated in the "Private" column as using private questions or answers.
| Domain | Evaluation marked in the Private column | Weight in the overall score |
|---|---|---|
| Knowledge work | AA-Briefcase | 15% |
| Knowledge work | AutomationBench-AA | 5% |
| Coding | FrontierCode 2.0 | 15% |
| Science | CritPt v2 (hard problems) | 10% |
| General capability | AA-Omniscience v1.1 | 15% |
| Total | 5 evaluations | 60% |
The five weights add up to 15 + 5 + 15 + 10 + 15 = 60%. On the other hand, the newly added Terminal-Bench-Science and Terminal-Bench 4.0 are not treated as private in the Private column of the official methodology. The same applies to GDP.pdf and HLE-Diamond. Of the 25% allocated to the science domain, only CritPt v2's 10% counts as private; the remaining 15%, held by Terminal-Bench-Science, does not. The 60% figure for the overall score must therefore be distinguished from whether each individual benchmark is public.
The newly added FrontierCode 2.0 evaluates whether code changes written by AI reach a quality that can be merged into real development projects. It uses 132 tasks, each run three times. The agent execution environment, in which the model works while operating tools, is "Stirrup." Scoring uses task-specific tests and a rubric, with the LLM judge being GPT-6 Luna at the medium setting. Humans do not grade every change directly; a submission scores zero if it fails a mandatory requirement or if the model is detected trying to look up the correct answer, and otherwise scores according to the weighted rubric items.
The other new evaluation, Terminal-Bench-Science, targets practical tasks involved in scientific research. It runs 70 tasks with "mini-swe-agent," three times each, and averages "pass@1," the probability of success in a single attempt. A run counts as a success only if it passes all tests, and the grading environment is isolated from the environment in which the model works. A cap of 1,000 steps also applies to the number of actions.
According to the Terminal-Bench-Science team's announcement, the fields covered include life sciences, physics, earth sciences, mathematics, and engineering. Rather than merely checking whether a model answers a question correctly, the benchmark verifies the outputs produced in real work, such as data analysis, simulations, mathematical proofs, and code, through separate tests. The initial results the team published on August 27, 2026 and the results Artificial Analysis announced in October were measured with different implementations, so they cannot be directly compared as results under identical conditions.
Existing evaluations were also changed to maintain difficulty and measurement precision. HLE-Diamond was updated to 1,000 questions: 500 reasoning and 500 knowledge questions. CritPt v2 removed 30 of the previous 70 problems, which seven models had all answered correctly in all five attempts, leaving 40 harder problems. AA-Omniscience also dropped saturated or already-public questions and corrected ambiguous items and errors. In other words, v5 scores reflect not only model capability but also the effect of changes to the evaluation questions themselves.
A 12-point or 26-point gap from the top, depending on whether model weights are open
According to Artificial Analysis's v5 announcement published on October 27, the leader at the time of the announcement was Claude Opus 5.5 at the max setting, with an overall score of 58. It was followed by GPT-6 Astra at max (57) and GPT-6.1 Sol at max (55), while Gemini 4 Argon at the high setting scored 50. Kimi K3.1 Pro scored 46, but its model weights had not been released at that point.
Taking the leader's 58 points as the baseline, the gap to Kimi K3.1 Pro is 12 points. However, DeepSeek V4.1 Flash 1017 and GLM-5.3, whose model weights are public, both scored 32, widening the gap to the leader to 26 points.
| Comparison | Model weights as of Oct. 27 | v5 score | Gap from leader (58) |
|---|---|---|---|
| Kimi K3.1 Pro | Not released | 46 | 12 |
| DeepSeek V4.1 Flash 1017 (max) | Released | 32 | 26 |
| GLM-5.3 (max) | Released | 32 | 26 |
This table uses the same overall index as of the October 27 announcement and subtracts each model's score from the leader's 58. The max setting is used for Opus, DeepSeek, and GLM, while Kimi uses the setting from the announcement. "Released" here means that the model weights are available; it does not compare open-source license terms. Whether the gap to the top model is 12 or 26 points depends on how one defines the release status of the models being compared.
That said, a gap in overall score does not imply the same performance gap in every use case. For example, Terminal-Bench-Science success rates were 63% for GPT-6 Astra, 59% for Claude Opus 5.5, and 58% for GPT-6.1 Sol. Kimi K3.1 Pro reached only 34%, and the open-weights DeepSeek just 13%. Yet on Terminal-Bench 4.0, Kimi took first place with 67%. As of October 27, no model led all nine evaluations, and these individual benchmark success rates are a different metric from the overall index score.
The overall ranking also shifted the day after the announcement. In results Artificial Analysis published on October 28, Gemini 4 Carbon at the high setting scored 59 and became the new leader. However, the model is at a stage of rollout to selected users and is not generally available. The gaps in the table above are based on the October 27 leader's 58 points, not a comparison with the top model as updated on October 28.
The gap between Kimi and the top model was 3 points in the previous v4.3 but widened to 12 points in v5. Because the evaluation items and difficulty have changed, however, this gap cannot simply be read as a decline in model performance. Likewise, the ranking alone does not allow one to conclude anything about evaluation questions leaking into training data, cheating, or the relative strength of reinforcement-learning infrastructure. Nor does a 9-point difference represent a ratio of capability or a statistically significant difference. When actually choosing a model, it is necessary to check results close to the target task and the execution conditions, not just the overall ranking.
The same model can differ greatly in score and cost depending on reasoning settings
When comparing AI model performance, the setting that determines how much computation is devoted to reasoning matters as much as the model name. For example, Claude Opus 5.5 scored 35 at the low setting, at a cost of $0.98 per task. At the max setting, it rises to 58, but the cost also increases to $10.04. That is, for the same model, the reasoning setting produced a 23-point gap in overall score and roughly a tenfold difference in cost. Specifying only the model name does not determine the performance or cost one will actually get.
The scores and costs of the top three models at the time of the October 27 announcement are as follows.
| Model and reasoning setting | v5 score | Cost per task |
|---|---|---|
| Claude Opus 5.5 (max) | 58 | $10.04 |
| GPT-6 Astra (max) | 57 | $6.24 |
| GPT-6.1 Sol (max) | 55 | $1.27 |
The overall score gap between Opus 5.5 and Sol is only 3 points, yet the cost per task differs widely, at $10.04 versus $1.27. Even among models with similar performance, the amount of work that can be processed on the same budget may differ. Among the top 10 models, the cheapest, Terra at max, cost $0.96 per task and the most expensive, Fable 5.1 at max, cost $13.04, a difference of more than 13 times. This does not mean the two models scored the same; it shows that costs vary widely even among top-ranked models.
The number of tokens models output also rose in this evaluation. The median output token count increased from 35,000 to 59,000. In particular, the four agentic evaluations account for 55% of the overall score weighting but about 90% of the median output tokens overall. Evaluations that require long-running processing or multi-step work make up most of the token consumption. However, the change in cost cannot be explained solely by the increase in agentic evaluations.
To calculate cost, Artificial Analysis combines the token counts reported by each model's provider with measured typical cache hit rates. The costs in the table are therefore averages for processing Intelligence Index tasks, not costs per successfully completed task. Nor do they guarantee that real Japanese-language business work can be processed at the same cost. The evaluation mainly targets English and text, and multilingual capability and similar traits are measured by separate evaluations.
Furthermore, the agent environments that run the models and the tools available differ for FrontierCode 2.0 and Terminal-Bench-Science. Benchmark scores are results obtained under conditions that include these execution environments, and do not show model capability alone. When deploying a model in real work, it is important to align the reasoning settings, tools, and execution infrastructure to be used, and then measure the success rate and cost for the required tasks.
How reliable are private evaluations, and how should long-term model progress be measured?
Increasing the number of private evaluation questions brings both benefits and challenges. The Decoder's analysis points out that adopting private tests makes it harder for outside researchers to discover errors or flaws in the questions. Keeping questions and answers private may curb over-optimization to known problems, but it reduces opportunities for third parties to verify the questions themselves. Both sides must be considered to secure a benchmark's credibility.
The correction of ambiguous and erroneous questions in AA-Omniscience this time also shows that the quality of evaluation questions affects scores. However, making questions private does not fully prevent contamination of training data. Conversely, private questions do not necessarily make a benchmark as a whole unverifiable. What matters is how much of the evaluation's composition, scoring criteria, and execution conditions is public, and how far third parties can confirm the validity of the results.
Comparing Intelligence Index versions also requires caution when assessing long-term model progress. In the official methodology for "Progress Index v1", Artificial Analysis states that because the Intelligence Index changes its evaluation items with each version, it is not intended for long-term comparison using scores from different versions. The Intelligence Index is a metric for comparing models available at a given time under common conditions. Examining how much AI capability has improved over the long term, while the evaluation targets and question difficulty change, requires a different measurement method.
That is what Artificial Analysis introduced with Progress Index v1, which uses Item Response Theory. This method jointly estimates the difficulty of evaluation questions and the ability of models, and statistically links evaluations from different periods using shared questions and model results as clues. This makes it possible to compare changes in capability even when the questions used for measurement are replaced. Rather than simply subtracting old-version scores from new-version scores, it makes clear what shared information was used to connect the evaluation scales.
Intelligence Index v5 has evolved toward more practical tasks, such as code development and scientific research, while increasing the weighting of private evaluations. However, changes in the overall ranking alone do not reveal which uses each model excels at or how much it would cost in actual work. When selecting a model, one should compare per-use-case results and costs within the same version, and when examining long-term capability progress, use a separate metric such as the Progress Index. Ultimately, verifying the success rate and cost for the target task under the reasoning settings and execution environment one can actually use is what allows benchmark results to translate into appropriate model selection.