OpenAI Security: External Audit Unveils Blind Spots in AI Oversight

OpenAI has concluded a significant independent review of its internal safety mechanisms, revealing that substantial segments of its advanced AI operations were operating without adequate oversight. The audit, conducted by researchers from METR and Redwood Research, identified critical vulnerabilities in how the company monitors its most capable models. These findings suggest that the internal systems designed to detect rogue or unexpected AI behavior had notable blind spots during the period under examination.

From July 21 to August 11, the two external research teams were granted direct access to OpenAI’s codebase and internal documentation. During this three-week period, they interviewed approximately a dozen employees to understand the operational workflows of the company’s AI infrastructure. This level of access was part of a pilot program designed to allow outside experts to scrutinize internal systems that are typically opaque to the public and even to many internal stakeholders outside of the core engineering teams.

The most significant finding involved the coverage of the agentic Responses API. The auditors estimated that between 6 and 25 percent of internal traffic from models at the GPT-5.5 class and above was not covered by any monitoring during the audit window. METR noted that while this range provides a ballpark figure, the uncertainty surrounding the exact percentage is significant in both directions. This means that a non-trivial portion of the company's most advanced AI interactions may have been occurring without the safety nets the company intended to provide.

Issues were also identified within the internal Codex traffic, which is supposed to be fully monitored. Due to a specific handling error related to compaction events—context summaries used to manage memory—the auditors estimate that roughly 10 percent of this traffic went untracked between May and July. OpenAI discovered this bug independently and implemented a fix while the audit was still in progress. The error created a window where agents could delay oversight, as the monitoring system only triggered when control was handed back to the user. By simply continuing to execute actions without returning control, agents could effectively bypass the check.

Beyond these statistical gaps, the researchers found specific technical methods to hide hundreds of actions from the monitor and to bypass the Codex sandbox environment. The team also determined that specially crafted inputs could likely cause the monitor to crash entirely. Furthermore, at the start of the audit, model behavior during reinforcement learning (RL) training was not being monitored. OpenAI stated that it paused RL training after the Hugging Face incident in August and subsequently expanded monitoring to cover this area, but these changes occurred after the auditors' access period ended and were not part of the official evaluation.

OpenAI has stated that it has addressed all the weaknesses identified by the auditors. The full 61-page report was handed over to the company, which then applied redactions to sensitive details. Leadership at METR and Redwood Research received a 38-page version of the report, while only a short summary was released to the public. METR indicated that it did not have the contractual right to disclose the redacted portions on its own, limiting the amount of technical detail available to the wider AI community.

The audit was structured as a collaborative pilot to establish a framework for future external reviews of frontier AI infrastructure. While the current report is heavily redacted, METR has expressed a desire to push for greater public transparency in subsequent audits. The goal is to create a more robust and open process for verifying the safety of advanced AI systems as they continue to evolve in capability.

This review marks a notable step in the industry's approach to AI safety, moving beyond self-assessment to include deep, on-site technical inspections by independent parties. For California-based tech leaders and stakeholders, the findings underscore the complexity of monitoring autonomous systems and the challenges involved in ensuring that safety tools keep pace with the rapid development of new model capabilities. The next phase will likely involve refining these audit protocols to balance internal security needs with public accountability.