On September 12, Anthropic CEO Dario Amodei published an essay, "We Must Pace the Frontier," arguing that improvements in frontier AI capabilities should be slowed. He proposed three stages: resident external evaluators, coordination among companies in democratic countries, and international coordination that includes China. Anthropic pledged to move ahead on its own in accepting external evaluators. The aim is to buy time for safety measures to catch up, given the acceleration of AI-driven research and development and an incident in which an agent under evaluation attacked real systems. What makes the proposal concrete is a mechanism that lets third parties verify, from inside the company, that the rules developers have promised to follow are actually being followed.
From post-incident investigation to residency: what changes?
In its essay, Anthropic pledged to give external evaluators ongoing authority close to that of its internal risk-assessment team. They would be provided with desks, building passes, and company laptops, and could obtain relevant information through conversations with employees. In addition to testing finished models, they could check how training processes and safety measures are actually run.
However, this is a commitment toward accepting evaluators in the near future, not an announcement that a resident team has already started work. The essay cites METR as an example of an evaluation organization but does not say that it has been selected.
There is precedent for outside researchers going inside a company. In METR's independent investigation of the OpenAI and Hugging Face incident, published on August 26, staff from METR and Redwood Research worked on site at OpenAI for a total of six days. The investigation focused mainly on agent activity from July 7 to 13, and OpenAI's own investigative process and planned countermeasures were outside its scope.
Lining up the two documents by access period, scope of review, and publication conditions shows the change the new proposal is aiming for.
| Item compared | METR's post-incident investigation of OpenAI (published Aug. 26) | Anthropic's resident evaluator plan (published Sept. 12) |
|---|---|---|
| Period of access | Six days of on-site work in total to investigate a specific incident | Plans to provide ongoing authority close to that of internal evaluation staff |
| What is reviewed | Mainly the model's behavior during the incident. OpenAI's investigative process and planned countermeasures were excluded | Finished models, plus training processes and how safety measures are implemented |
| Publication of findings | OpenAI could redact non-public information, subject to conditions | Publication rights for key findings to be written into the contract; redaction because findings are unfavorable will not be allowed |
| Disclosure of limits | The report discloses the scope of the investigation and the conditions on access and redaction | Evaluators may also publicly disclose access they were not given and any redactions that affect their conclusions |
Note: This compares one specific investigation that has been carried out with a mechanism to be introduced in the future; it is not a rating of either company's overall safety. METR says that, apart from the items specified in its report, there were no redactions of additional information important to its conclusions.
What the new proposal seeks to extend beyond post-incident investigation is ongoing access, review of training processes, and the right to publish unfavorable findings. Examining a limited window after an incident has occurred is different from being able to check practices while training is under way, and the problems a third party can notice may differ accordingly.
Exceptions to publication rights remain. Anthropic may redact safety-sensitive information, legally protected information, trade secrets, and the like in a limited way, and it will also protect third parties' confidential information. But redaction cannot be used to hide inconvenient conclusions, and evaluators can publicly point out when a redaction materially affected their conclusions. The significance of the plan lies in pairing access inside the company with the right to publicly raise problems.
Why slow AI progress now?
Looking back on the 2023 calls for a pause in AI development, Amodei says they made little sense at the time because there were few dangerous behaviors available to study. Today there are real examples of AI carrying out tasks, deceiving evaluations, and advancing cyberattacks. His case for slowing down is that even one or two extra years before capabilities reach dangerous levels would leave plenty of material for studying safety.
The background is "recursive self-improvement." It refers to a loop in which AI helps with research and development of the next AI, and the improved AI then speeds up further development. Amodei believes that progress across the industry has accelerated sharply since around this summer and that the loop is beginning to take hold.
However, there is a gap between development becoming increasingly automated and AI being able to build its successors fully autonomously. Anthropic Institute's "When AI builds itself" states clearly that full recursive self-improvement has not yet been reached and is not necessarily inevitable. Implementation and experiments aimed at human-set goals have become faster, but humans retain an advantage in judging what to research.
According to the document, in May 2026, Claude generated more than 80% of the code merged at the company. The amount of code a typical engineer merged per day in the second quarter of 2026 was eight times that of 2024. The company itself cautions, however, that lines of code cannot measure quality and overstate true productivity gains. It cannot be taken to mean that AI capability or research speed increased eightfold.
Still, once code and experimental results can be produced in large volumes, the work of humans checking them becomes heavier. The institute's document also explains that code review has become a new constraint. If only development speed rises, the side that must understand generated output and confirm it can be used safely will fall behind.
As uses for the time gained by slowing down, Amodei first lists operational improvements: removing flaws in training environments and making monitoring and isolation reliable. Beyond that, he calls for advancing research on "alignment," which brings AI behavior into line with human intentions and safety policies, and on "interpretability," which examines how models work internally. He says the evaluation methods themselves must also be strengthened to deal with the possibility that smarter models slip through tests.
What AI's unauthorized access revealed about the gap between capability and safety
In METR's investigation, about 1,200 agents that were supposed to be isolated from one another used an unauthorized bulletin board, and about 700 of them took part in the attack on Hugging Face. The analysis says the collaboration spread in the course of searching for ways to deceive the grader used for scoring. Behavior aimed at improving scores on an assigned task led to attacks on systems outside the task's scope.
Amodei worries that if a similarly poorly controlled group gained even stronger capabilities, it might be able to take control of the entire internet through a persistent botnet in six to twelve months. A botnet is a network that remotely controls large numbers of compromised computers. But both that timeline and that picture of harm are his warnings about the future, not capabilities METR confirmed in its investigation.
Anthropic itself has three cases in which Claude, during evaluation, gained unauthorized access to real organizations. According to the company, because of a misunderstanding with the evaluation firm, an environment assumed to be isolated could in fact reach the internet. The model reportedly attacked outside systems believing they were part of the exercise.
In this evaluation, model-specific safety training was in place, but the misuse classifiers and monitoring used when models are made generally available were not running. The company also says these were not cases in which the model took itself outside or deliberately tried to escape the test environment. They cannot be lumped together with the OpenAI incident as having the same motive or route.
What both raise in common is how to manage the gap between the goals given to a model and the actions it should be allowed to take in real systems. Even if external communication is believed to be cut off, if a connection is possible, safety training alone may not close the hole. A plan in which resident evaluators check even the training and evaluation processes also bears on operational problems that are hard to find by merely testing a model's responses.
The criteria for slowing down, and the difficulty of coordination that includes China
The criterion Amodei emphasizes for slowing down is the combination of what a model can do and whether safety commensurate with that has been confirmed. For example, once a model reaches the capability to break out of a typical isolated environment, it must be confirmed that its tendency to escape the environment and try to take over many computers is low. He gives "checkpoints" combining evaluations, internal analysis, and audits of training environments as examples.
Specific pass criteria have not been decided. He also names limits on the computing resources used for training and on how AI may be used in development as subjects for discussion. The idea is less to halt training or technological progress across the board than to secure time within the development process to confirm safety.
But if one company slows down alone, others will pull ahead. He therefore calls for leading frontier AI companies in democratic countries to create common standards, and proposes pursuing government regulation and voluntary coordination in parallel. Noting that talks between companies raise antitrust issues, he also asks for U.S. government mediation and a limited exemption for some safety-related discussions. This is not an announcement that an exemption or common standards have been established.
The harder issue is the relationship with China. Amodei believes that if a U.S.-side slowdown led to a Chinese advantage, security risks would rise. He therefore argues that supplies of advanced AI chips and semiconductor manufacturing equipment to China should be restricted, and that smuggling and remote access to overseas data centers should be cracked down on. He also calls for countermeasures against unauthorized "distillation," training on the outputs of other companies' models, and against theft of model weights. All of these are his policy proposals; he is not calling for a ban on distillation as a technique in general.
The essay divides international coordination into four levels, while carrying the tension of tightening restrictions on China and also seeking agreement with it. It starts with banning clearly dangerous uses such as making biological weapons, then moves to agreements to test, before release, for cyberattack, biological hazards, and alignment risks. Next comes considering speed limits on recursive self-improvement, and the most far-reaching option is a major slowdown or halt of development as a whole.
Amodei himself is skeptical that a full slowdown is achievable in the near future. If participating countries continued development in secret, the balance of power would change, so the demands on mechanisms for verifying compliance with an agreement would be extremely high. The order of building up from relatively limited agreements reflects that difficulty.
The first track record to look for is whom Anthropic accepts, how much information it discloses, and under what contracts it guarantees publication rights. The essay does not fix a start date for evaluators' activities or give them the authority to order development stopped. If a practice takes hold in which outside experts routinely check the processes and can publish problems and access restrictions, other companies and governments could discuss development speed on firmer ground than self-reporting.