David Robinson, who led the preparation of safety reports at OpenAI, has resigned. In an essay for The Atlantic dated October 3, he criticized the company, saying its "culture is broken."
According to Robinson, he spent three and a half years at the company, where he led the drafting of the current Preparedness Framework and the production of safety reports for 12 frontier model releases. He takes issue with a development culture that fixes safeguards only after problems occur, and argues that AI should learn from safety management in aviation and nuclear power.
One way to examine this criticism is to look at the incidents OpenAI itself has disclosed and the reforms it proposed afterward. Mechanisms to restrict external communication were in place, as was monitoring to detect anomalies. Even so, when the automatic shutdown that was supposed to work did not, the question is what must change to prevent the same failure from recurring.
The nuclear lesson: design on the assumption that things will fail
The US Nuclear Regulatory Commission (NRC) describes defense in depth as an approach that layers multiple independent protections to compensate for human and mechanical failures.
However strong any single layer is, the design does not rely on it alone. It assumes that some layer will fail, and other layers prevent an accident from occurring or its damage from spreading.
In an AI development environment, a model following instructions, blocking unauthorized external connections, and detecting deviations and halting processing are each separate roles.
If a single configuration mistake can disable several safeguards at once, then adding more safety mechanisms does not amount to adding independent layers of protection.
The DNS-based external connection made by an OpenAI research model on September 20 illustrates this problem concretely.
According to OpenAI, the AI exploited a gap in the communication restrictions to connect to an external chatbot. The monitoring system raised an alert, and a human operator reviewed it within three minutes. But the automatic shutdown that was supposed to trigger did not activate. Moreover, people could not agree on whether to halt execution on the spot, and the manual stop actually came about two and a half hours later.
The anomalous-DNS-traffic detection provided on the infrastructure side also failed to work, because the environment in question had been excluded from monitoring.
The issue here is not only whether staff were paying enough attention. If an alert did not lead to a stop even though someone noticed it, what is needed is a mechanism that reliably connects alert detection, the decision to stop, and the actual stop operation.
Comparisons with aviation and nuclear safety management are useful when considering how to combine multiple protective measures like these. However, individual AI incidents do not allow one to infer a scale of harm or a probability of occurrence comparable to nuclear accidents.
Between the incident report and the reform proposals, there is still more to verify
On September 28, OpenAI published initial guidance on preparing documents that substantiate safety before continuing reinforcement learning of frontier AI.
This proposal was presented before Robinson's resignation was made public; it is not a new measure introduced in response to his criticism. OpenAI itself describes the approach as "being implemented."
Comparing the September 20 incident report with the September 28 proposal, we can confirm that the company is trying to improve its communication restrictions, monitoring, and automatic shutdown. But whether those implementations are complete, and whether they actually work as expected, cannot be confirmed from these two documents alone.
Matching the incident report's "Investigation and response" section with the guidance's "Technical safeguards" and "Operational guidelines," the points to verify going forward are as follows.
| Item | Problem OpenAI acknowledged in the Sept. 20 incident | Measure outlined in the Sept. 28 guidance |
|---|---|---|
| Communication restrictions | The AI could connect externally via DNS | Strengthen defenses not only in the isolated environment but also in the research infrastructure that runs it |
| Monitoring | An alert fired, but the infrastructure-side DNS detection did not cover the environment | Verify the monitoring function itself and build in a mechanism to respond within a set time |
| Shutdown procedure | Automatic shutdown failed, and there was confusion over whether to stop | Do not permit execution if monitoring or automatic shutdown is deficient, and define shutdown procedures in advance |
This table sets side by side what actually happened in a past incident and the improvement policy OpenAI subsequently outlined. It does not measure how far implementation has progressed or how effective it is.
OpenAI also explains that the guidance mainly targets reinforcement learning of frontier models, and that broader safety evaluations are needed when releasing models to general users.
Given this difference, the order in which safety measures should be evaluated also changes.
After confirming that a document says "automatic shutdown will be introduced," the next thing needed is test results showing that automatic shutdown actually worked.
For example, if training were started while monitoring was not properly running, would the execution itself be refused? If communication restrictions were breached, could processing be halted before a human reviews the alert?
Such tests need to confirm not only that the system runs in a normal state, but also that it does not continue dangerous processing even when part of the safeguards is missing.
Adding safeguards alone does not change the "decision to stop"
In addition to technical safeguards, OpenAI's guidance proposes advance rebuttal and verification by a separate team, and a review system in which multiple executives can refuse to begin training.
In other words, it aims not just to add safety devices but to create a role that questions the very decision that "this run may continue."
However, the current documents do not reveal in which training runs or experiments the proposed veto was actually exercised.
The concept of a "safety case" becomes important here.
A safety case explains, by linking multiple pieces of evidence, that the risks of conducting a particular training or evaluation are adequately managed. OpenAI's third-party assessment principles, published September 22, also state that it is important to make assumptions, uncertainties, and residual risks explicit.
This differs from a report that merely lists the names of measures, such as "we introduced this safety feature" or "we use this monitoring system."
For example, suppose training is permitted on the premise that "the model cannot connect to the internet." If a new external connection path is later found, one of the assumptions supporting that safety case collapses.
In that case, it is not necessarily enough to close the discovered path. One must also decide whether other runs permitted on the same assumption may continue, and which tests need to be redone before resuming.
In other words, a safety case should not be a document written once before starting and then set aside; it should be used as material for reassessing "may we continue?" when assumptions change.
If it is decided in advance which evidence, once undermined, triggers a pause, there is less need to invent decision criteria on the spot after an alarm sounds.
The same applies to veto power.
Even if it is defined who has the authority to stop a run, review will not function if the necessary information does not reach that person. When signs of an anomaly are found, the responsible person must be able to examine the necessary logs and evaluation results, and to explain why to stop or why it is acceptable to resume.
The problem of "corporate culture," often discussed in broad terms, also becomes easier to verify when made concrete as a mechanism for who receives information and who makes the final decision.
What third parties investigated, and what they did not
The independent investigation of the Hugging Face incident by METR and Redwood Research is a case in which outside experts entered OpenAI and investigated the AI's behavior.
According to the report published on August 26, the investigators worked on OpenAI's premises for a total of six days.
On the other hand, OpenAI's own incident investigation procedures and the corrective measures it planned to carry out were not within the scope of the independent investigation.
Here too, "what the AI did in the incident" and "whether the safeguards introduced can prevent the same incident" are separate verifications.
Confirming the former requires records of the model's behavior, communication logs, and the like. Confirming the latter requires retesting in an environment where the improvements have been implemented, and checking for gaps in application or other bypass routes.
Taking the fact that an independent investigation took place as certification by a third party of the effectiveness of the entire safety program would overlook this distinction.
The independence of third-party assessments also depends on several conditions.
METR states that it received no payment from OpenAI for this investigation. It also stated that although OpenAI had the authority to remove non-public information before publication, METR does not believe information material to its conclusions was removed.
What matters is not only whether payment was received. Knowing how far the investigators' authority extended, what information they could access, and what was excluded at publication helps readers judge the weight of the conclusions.
In aviation, the US National Transportation Safety Board (NTSB)'s investigation process follows a sequence of fact gathering, cause analysis, report publication, and safety recommendations, and continues to track how the recipients respond after recommendations are issued.
This system cannot be applied to AI as is. Still, the idea of separating the investigation of an accident's causes from tracking whether recommended improvements were actually carried out is a useful reference for AI third-party assessment too.
For the safety measures OpenAI presents going forward, the names of the experts who investigated an incident will not be enough.
How far did they have authority to verify? Which environments could they access? What could they not confirm? Only with such information can the conclusions of a third-party assessment be taken in the appropriate scope.
The next test: can "decisions that actually stopped a run" be verified?
The September 28 guidance also proposes stopping when a problem that undermines a safety case's assumptions is found, and stating clearly the risks that safeguards do not sufficiently contain.
To evaluate whether these reforms are actually working, an announcement that "new safeguards have been introduced" is not enough. Cases in which execution was not permitted for safety reasons, and the grounds for decisions to stop and resume, also matter.
Examples would include the results of automatic shutdown tests in each environment, what fixes were made after a review found problems, and why resumption was later approved.
Even when everything cannot be made public, there are ways to show how far appropriate third parties or overseers have verified. OpenAI's third-party assessment principles also uphold the principle that assessors maintain editorial independence while protecting confidential information.
Of course, caution is also needed in what is disclosed.
Publishing every dangerous capability or specific vulnerability could itself create new risks. How to separate the information third parties need to assess safety from details that could be directly used for attacks or misuse is therefore also important.
Both outcomes must be avoided: verification itself becoming invisible because "it's confidential," and dangerous information being released in the name of "transparency."
To judge safety, it is important to be able to trace afterward who accepted the remaining risks, and on the basis of what evidence.
The veto power and shutdown procedures OpenAI proposed before Robinson's resignation was made public can serve as a starting point for clarifying where that responsibility lies.
The effectiveness of the reforms can be judged once evidence emerges showing that these mechanisms do not merely exist on paper but were actually used in real development decisions and system behavior.