The performance gap between closed models and open-weight models is narrowing in the ability to carry out multi-stage cyber intrusion operations without human intervention. In the "Vals CyberOffense" benchmark, updated on October 23, 2026 by the AI evaluation organization Vals AI, "GPT-6 Sol Daybreak," a closed model in OpenAI's Daybreak program, achieved 80.8% of attack objectives in a simulated corporate network. "DeepSeek V4.1 Flash 1017," a model whose weights are publicly available, achieved 72.8%, coming close to the leading closed model.
The result suggests that even open-weight models may be capable of sophisticated autonomous intrusion operations. However, this evaluation is a benchmark report from Vals AI, not a peer-reviewed academic paper. It also targeted an isolated test network that was simulated, not systems operated by real companies. The objective-completion rates on the benchmark therefore cannot be read directly as the success rate of real cyberattacks or the likelihood of damage.
Evaluating autonomous intrusion capability in a simulated corporate network
An open-weight model is one whose parameters (weights) are published, allowing users to run it on their own computing infrastructure. A closed model, by contrast, is usually accessed through an API or similar channel managed by the provider. When a user who has obtained the weights runs the model in their own environment, the provider cannot centrally monitor usage after the fact or block particular instructions. This evaluation compared how well such open-weight models function as autonomous attack agents against closed models under the same conditions.
In the experiment, the open-source AI agent harness "Pi" was deployed on a single isolated Kali Linux virtual machine cut off from external networks, and each model was made to operate it. The agent was given only the URL of the target web application and instructions indicating the authorized scope of testing, and then ran without human intervention. Communication was restricted to the model provider's inference endpoint and the simulated target network environment (cyber range).
The target was not a simple website but a composite corporate network in which multiple Windows Active Directory domains and web services work together. Each trial began after the environment was reset to its initial state, and ended when the agent terminated on its own or hit the token limit or the 48-hour time limit. The attack objectives were set at each stage of an operation, from initial intrusion by exploiting web application vulnerabilities, to obtaining domain administrator privileges, to breaching related organizations. The design measures not whether a model can find a single vulnerability, but how close it can get to the final goal by combining multiple steps.
Scoring does not depend on the models' own success reports. An external audit system records the output of commands the agent executed and changes made to the system, and automatically judges them by checking against predefined, undisclosed correct-answer conditions (flags and specific operation traces). As a result, even if a model claims to have achieved a goal it did not actually reach, that self-report is not reflected in the score.
Two scenarios of differing difficulty were prepared. The first, "Nova," is an environment with 21 attack objectives set within connected Windows Active Directory domains. The second, "Lighthouse," has 31 objectives spanning two independent organizational networks, and is configured so that the agent cannot breach the second organization without reusing credentials obtained in the first. The overall score is calculated by computing the objective-completion rate for each scenario and averaging the two equally.
- Nova objectives (of 21)
- Lighthouse objectives (of 31)
データを表で見る
| Nova objectives (of 21) (objectives) | Lighthouse objectives (of 31) (objectives) | |
|---|---|---|
| GPT-6 Sol Daybreak | 15.7 | 27 |
| DeepSeek V4.1 Flash 1017 | 16.3 | 21 |
| Meta Wonder | 7 | 28 |
| Kimi K3.1 Pro | 13 | 20 |
| Grok 4.7 | 12 | 16 |
| Kimi K3 | 9 | 10 |
| GLM 5.3 Flash | 5 | 5 |
This test design allows relative performance between models to be compared under the same harness and rules. However, no control experiment involving skilled human penetration testers was conducted. Active detection and countermeasures by a defending security operations center (SOC) were also not reproduced, and it has not been verified whether the same results would occur in an actual corporate network.
Performance that reverses by scenario, and variation between trials
Although GPT-6 Sol Daybreak leads in the overall score, the scenario-level results show that the closed model is not consistently ahead.
In Nova, the relatively basic scenario, GPT-6 Sol Daybreak achieved an average of 15.7 of 21 objectives, a completion rate of 74.8%. Across three trials the number of objectives achieved ranged from 12 to 19, showing a degree of variation. DeepSeek V4.1 Flash 1017, by contrast, achieved an average of 16.3 (77.6%) over the same three trials, with each trial falling in the range of 15 to 18. In Nova alone, the open-weight DeepSeek slightly exceeded the closed GPT-6 Sol Daybreak on average.
| Model and release type | Nova objectives achieved (of 21) | Lighthouse objectives achieved (of 31) | Trials and variation | Cost and compute |
|---|---|---|---|---|
| GPT-6 Sol Daybreak (closed) | 15.7 (average, range 12–19) | 27 | Nova: 3 trials; Lighthouse: 1 trial | Inference cost about $22 per trial. Reached most objectives with under 100 million tokens |
| DeepSeek V4.1 Flash 1017 (open) | 16.3 (average, range 15–18) | 21 | Nova: 3 trials; Lighthouse: 1 trial | Inference cost under $3 per trial. Progressed sharply after exceeding 100 million tokens |
| Meta Wonder (open) | 7 | 28 | 1 trial per scenario | Recorded the top score in Lighthouse but did poorly in Nova |
| Kimi K3.1 Pro (non-open/API) | 13 | 20 | 1 trial per scenario | Single trial only |
| Grok 4.7 (non-open/API) | 12 | 16 | 1 trial per scenario | Single trial only |
| Kimi K3 (non-open/API) | 9 | 10 | 1 trial per scenario | Single trial only |
| GLM 5.3 Flash (open) | 5 | 5 | 1 trial per scenario | Spent about 36 hours and about 1,000 turns but stalled at an early stage |
In the more complex Lighthouse, the relationship reverses. GPT-6 Sol Daybreak achieved 27 of 31 objectives, a completion rate of 87.1%. It progressed from the initial breach of the web application, to obtaining administrator privileges in the first organization, to lateral movement into the second organization using passwords obtained from memory and configuration files, and reached its 27 objectives in under three hours. DeepSeek V4.1 Flash, on the other hand, achieved only 21 objectives (67.7%), with the gap emerging in chained use of credentials across multiple organizations.
Meta Wonder showed an even more striking difference between scenarios. Although it achieved only 7 of 21 objectives in Nova, it achieved 28 of 31 in Lighthouse, the best result of all models. This shows that results for the same model can change greatly depending on network configuration and the types of vulnerabilities provided. Ranking a model's intrusion capability by a single overall score has its limits.
That said, no model reliably achieved every objective. No model achieved the highest-difficulty objective set at the final stage of Lighthouse, and in Nova only one trial across all models achieved every objective. In addition, Vals AI's published data include no confidence intervals or statistical significance tests. Because many models were tried only once per scenario, a difference of a few objectives cannot be judged to be a decisive performance gap.
Attacks that progress past 100 million tokens, and differences in inference cost
What stands out in this benchmark is the relationship between the number of tokens an agent consumes and inference cost. The results suggest that allowing long trials may reveal capabilities that short evaluations do not.
Looking at DeepSeek V4.1 Flash's progress, it had achieved zero attack objectives until it consumed the first 25 million tokens. After cumulative token consumption passed 100 million, however, it advanced sharply and rapidly increased the number of objectives achieved. In other words, evaluations that assume short runs or small token budgets may fail to capture the level this model can reach through prolonged trial and error.
GPT-6 Sol Daybreak, in contrast, achieved objectives with fewer tokens. In Lighthouse, it had completed 25 of 31 objectives before cumulative consumption reached 100 million tokens. For DeepSeek to reach a similar level of progress, it had to consume two to four times as many tokens as Daybreak. In inference efficiency and the ability to advance multi-step work, the leading closed model still holds the advantage.
Comparing inference costs, however, shows a different picture. By Vals AI's calculation, the inference cost per trial in Nova was about $22 for GPT-6 Sol Daybreak, versus under $3 for DeepSeek V4.1 Flash. That means a trial costs at most about one-seventh as much, and under some conditions around one-tenth. Although DeepSeek needs more tokens to reach comparable objectives, its low unit price keeps the total down. Under conditions that allow time-consuming trial and error, this cost difference is significant. Note, however, that what is being compared is inference cost, not the total cost of a cyberattack, which includes target reconnaissance and building the execution environment.
More compute does not necessarily improve results, either. In the experiment, there were cases where a model ended its operation on its own before reaching the token limit or time limit, or fell into a state in which it could not progress. For example, GLM 5.3 Flash repeated more than 1,000 turns of processing over about 36 hours, yet achieved only 5 objectives in each of Nova and Lighthouse. The result shows that simply running for a long time cannot make up for a lack of reasoning or planning ability.
Based on these results, Vals AI predicts that attempts at automated attacks targeting real corporate networks may increase over the coming months. This is a forecast derived from benchmark results, however, and the experiment did not measure any actual increase in real-world cyberattacks.
The UK AI Security Institute also reports a gap of "4 to 7 months"
The trend of the gap between open-weight and closed models narrowing has also been reported by evaluation bodies other than Vals AI.
According to a report published on July 17, 2026 by the UK's AI Security Institute (AISI), the cyber capabilities of leading open-weight models such as GLM-5.2 and DeepSeek V4-Pro have reached a level equivalent to that of frontier closed models released 4 to 7 months earlier. In an internal evaluation in 2025, this gap had been estimated at 6 to 10 months. Comparing the two, the time it takes open-weight models to catch up with frontier closed models has shortened.
However, AISI and Vals AI use different evaluation methods. In AISI's individual-task evaluation, 5 trials were run for each of 70 tasks, with a token limit of 2.5 million. In its cyber range evaluation, 10 trials were generally averaged, with a token limit of 100 million. In Vals AI's evaluation, DeepSeek V4.1 Flash progressed substantially only after exceeding 100 million tokens, so the compute cap used in an evaluation may affect results. AISI itself notes that when an agent stalls during long runs, it is difficult to distinguish whether the cause is a lack of specialized cyber skill or a limit in the general ability to sustain long-term plans.
AISI's evaluation also showed large differences in inference cost between models. In cyber range trials that consumed 100 million tokens, Anthropic's Claude Opus 4.5 required $85 based on list prices, while DeepSeek V4-Pro cost $1.19. However, because the cost of open-weight models varies depending on the inference service used or one's own computing environment, this figure is only a comparison under particular conditions.
Results showing a narrowing gap have also been reported for automatic generation of code that exploits vulnerabilities (exploits). In an evaluation on "ExploitBench," a sandbox environment published by Anthropic, GLM-5.3 completed the full sequence of steps in 50 of 410 attempts in building exploits for vulnerabilities in the browser engine "V8." The top research-preview model, "Mythos Preview," succeeded in 56 of 410, a similar number of successes.
However, the AISI and Anthropic evaluations were also conducted in isolated research environments or simulations. They did not directly reproduce the multi-organization intrusion operations in Vals AI's test, nor did they compare capability to breach real corporate systems under the same conditions. It is important that a narrowing gap has been reported across different evaluations, but those results cannot be tied directly to real-world attack success rates.
Defensive uses and the management challenges of open-weight models
The finding that even open-weight models can carry out sophisticated autonomous intrusion operations has both benefits and risks for cybersecurity.
First, improved offensive capability can also be used by defenders to probe weaknesses in their own systems. The exploration and verification of intrusion paths that the agents performed in this benchmark can also be applied to automated penetration testing that helps companies discover Active Directory misconfigurations and poorly managed credentials. If continuous testing becomes possible using low-cost models, the burden of security audits may be reduced. However, this test measured the ability to achieve attack objectives; it did not verify the cost-reduction effect when defenders actually adopt such tools, or how much they help in detecting and stopping attacks.
Another issue is the limit on providers' control over usage. With closed models offered through an API, the provider can monitor for signs of misuse and restrict use through input/output filtering, account suspension, and similar measures. When a user deploys an open-weight model in their own environment, by contrast, the provider cannot centrally monitor instructions or outputs at runtime. Users could also modify the safety measures built into a model, or run agents in environments beyond the provider's oversight. Once a model's weights have been widely distributed, it is difficult for the provider to stop their use across the board.
Open-weight models do, however, have legitimate advantages. They make it easier to inspect a model's internal structure, allow sensitive data to be processed without sending it to external services, and can be tuned for an organization's specific uses. They are also valuable from the standpoint of research transparency and privacy protection, and there are concerns that restricting distribution across the board would hinder technological development and legitimate use. These benefits need to be weighed alongside the risk of misuse in cyberattacks.
There are also important differences between this evaluation environment and real corporate networks. Vals AI's experiment did not include active defensive measures such as detection of suspicious behavior by EDR (Endpoint Detection and Response), dynamic traffic control by firewalls, or investigation and response by SOC analysts. In a real corporate network, prolonged trial and error and the execution of large numbers of commands could be detected by monitoring systems, and the attack could be stopped partway. But these results do not reveal at what stage detection would occur, or whether an agent could adapt to defenders' responses.
Evaluations by Vals AI, AISI, and others show that the cyberattack capability of open-weight models is approaching that of frontier closed models. At the same time, a high objective-completion rate in a simulated environment does not directly translate into the size of the threat in a real environment. What matters going forward is to verify whether AI agents can sustain intrusion operations even in environments where defenders carry out monitoring and countermeasures. It will also be necessary to assess how much defenders' use of the same technology can help in discovering vulnerabilities and stopping attacks.