On November 16, 2026 (local time), OpenAI announced the first phase of "Embedded Evaluations," a program that brings four external evaluation organizations inside the company. Apollo Research, METR, Transluce, and Goodfire will gain access to models under development and records of internal use so they can verify AI alignment (consistency with human intentions and values) and safety measures. The setup lets outside evaluators engage more deeply with the development process than earlier external evaluations did, but the extent to which they can publish what they learn also changes what readers can use to judge the credibility of their reports. Of the three conditions Apollo says independent evaluation requires, the only one it has stated the contract meets is "meta-transparency," the ability to disclose the limits placed on an evaluation.
What the four organizations will evaluate, and how much internal access they get
According to OpenAI's announcement, external evaluators will be given access, as needed, to model checkpoints, source code, training and evaluation data, monitoring systems, and technical documentation. They will also be able to talk with the researchers and engineers involved. Evaluators who handle confidential information will work at OpenAI's offices on devices supplied by the company.
The focus of each evaluation differs by organization. Apollo Research will examine "sabotage," meaning attempts by an AI model to undermine the alignment or control mechanisms of itself or its successor models. Specifically, it will verify four claims in two stages: that the training process does not encourage sabotage, that evaluations and red-teaming do not provoke sabotage, that no such attempts occur during internal use, and that reasoning leading to sabotage can be detected.
METR will investigate inappropriate behavior that existing tests and monitoring systems missed, in what is the largest reinforcement learning run OpenAI has carried out to date. Transluce will verify the quality and coverage of alignment evaluations in post-training, and it intends to conduct an additional evaluation of about eight weeks jointly with OpenAI in early 2027. Goodfire will send interpretability specialists to analyze the inner workings of the models.
There will also be a mechanism for overseeing the results. The Safety and Security Committee (SSC) of the OpenAI Foundation's board will receive reports with no edits or deletions, and will make the final decision if disagreements between OpenAI and the evaluators go unresolved. According to Apollo, six of its staff will work inside OpenAI for four months and produce two public reports. The first will examine whether the training process encourages sabotage, and the second will cover evaluations, internal deployment, and whether reasoning leading to sabotage can be detected.
Apollo's three conditions for independent evaluation, and what the contract says
On its own blog, Apollo explains that ensuring the reliability of evaluation results without depending on trust in the developer requires evaluators to have three rights: sufficient access to relevant information; the right to disclose findings, even unfavorable ones, to appropriate outside parties; and "meta-transparency," the ability to explain to outsiders the restrictions placed on access and disclosure. Measured against these three conditions, the contract with OpenAI looks like this:
| Condition | What the contract says |
|---|---|
| Access | OpenAI may withhold some of the information subject to evaluation for reasons of technical constraints, security, or intellectual property. However, withholding because the information is unfavorable is not permitted, and OpenAI must disclose that information was excluded and give its reasons in writing |
| Disclosure | OpenAI may request removal of supporting evidence it judges to be highly sensitive intellectual property. However, it may not request removal because evidence is unfavorable, nor may it have Apollo's conclusion on each verification item removed |
| Meta-transparency | Evaluators may publish the contract terms, any withholding or delay of information and the reasons, and whether evidence was removed and how that affected the results, without prior approval from OpenAI |
Apollo describes the current pilot as a first step toward securing stronger rights on access and disclosure, and says it does not yet meet its own standard on those two. On meta-transparency, by contrast, it stated plainly that the contract meets the standard. In other words, evaluators do not have a guaranteed right to obtain all the information an evaluation requires, or to publish supporting evidence without restriction, but they can explain what they were unable to examine and what they were unable to publish.
In fact, part of the scope of accessible information is left to future negotiation. According to Apollo, the access specified for examining whether the training process encourages sabotage includes training code, datasets, reinforcement learning rollouts, and queries to intermediate checkpoints. For the other three verification items, the detailed arrangements on the access needed are to be agreed on later. Evaluators can ask for broader access, but there is no mechanism that obliges OpenAI to grant it.
There are also constraints on timing. OpenAI explains that, for findings that could pose safety or security risks, it may request a delay in publication under responsible disclosure principles to give it time to address them. Procedures for managing conflicts of interest, and for cooling-off periods when an evaluator works with multiple AI labs, are still being developed in consultation with the evaluation organizations.
The importance of meta-transparency also shows in how information from employees is handled. For claims obtained in conversations with OpenAI employees, evaluators cannot publish them without OpenAI's attestation, unless they can corroborate them independently. However, the fact that OpenAI declined to attest can be made public, as can an explanation of how that refusal affected the evaluation results. Even when supporting evidence is removed from a report for intellectual property reasons, evaluators can tell readers that it was removed.
Apollo points out that without such meta-transparency, outsiders could not tell why a report mentions no problems. An evaluator who simply could not access the necessary information might be read as having found nothing wrong.
Contractual safeguards, and what to check in the public reports
The contract also includes clauses meant to prevent OpenAI from interfering with the evaluations. OpenAI agreed not to refuse to provide information or contact with employees, or to request deletions from reports, because results are unfavorable. It also may not have evaluators' conclusions removed, and the SSC receives unedited reports. Given these mechanisms, one could argue that rating the contract as meeting only one of Apollo's three conditions is too harsh.
However, protecting an evaluator's conclusions is a separate matter from publishing the evidence behind them. OpenAI may request removal of evidence it judges to be highly sensitive intellectual property, even when that evidence bears directly on an evaluator's conclusions. Apollo itself acknowledges that readers of the public reports will not be able to check everything that underlies a conclusion. To judge a report's credibility, then, readers need to look beyond the conclusions to which information was withheld, which evidence was removed, and how much those constraints affected the results.
This can be checked in the first report Apollo publishes. If it specifies the scope of any withholding or evidence removal and states clearly that conclusions were unaffected, one could conclude that actual disclosure was fuller than the contract terms would suggest. If, on the other hand, the report only notes that removals or withholding occurred without explaining their effect on conclusions, readers cannot adequately judge the validity of the results from the published evidence alone.
The issue also matters when companies use third-party evaluations of AI vendors in procurement decisions. When Japanese companies review an evaluation report, three points are key: the scope of what was evaluated, whether any information was withheld, and whether evidence was removed and how that affected conclusions. If these are stated clearly, the report can be used with its limitations in mind. Apollo notes that the meta-transparency rights OpenAI has agreed to here could become a benchmark for contracts between other AI developers and external evaluators.
