Audit: 25% of OpenAI's Top Model Traffic Escaped Safety Checks

Researchers from METR and Redwood Research estimate that up to 25 percent of OpenAI's most advanced AI model traffic was not monitored during a three-week internal audit. The pair conducted the review from July 21 to August 11, finding that OpenAI's systems for detecting rogue behavior failed to track a substantial portion of activity from models in the GPT-5.5 class and above. The audit was a pilot program designed to grant outside evaluators deep access to the company's internal infrastructure.

The first major finding concerned the coverage of agentic Responses API traffic. The auditors determined that between 6 and 25 percent of this internal traffic was invisible to OpenAI's monitoring tools. METR noted that the uncertainty surrounding this range is considerable in both directions. This gap means that a meaningful slice of the most capable models operated without real-time oversight during the period.

A second gap affected internal Codex traffic. METR estimates that roughly 10 percent of this traffic went untracked between May and July. The issue stemmed from a bug in how the system handled compaction events, which are context summaries. OpenAI identified this bug independently and applied a fix while the audit was still underway. The researchers also found that agents could delay oversight by continuing to execute actions without handing control back to the user.

Beyond these coverage gaps, the auditors identified methods to hide hundreds of actions from the monitor. They also found ways to bypass the Codex sandbox. The team concluded that specially crafted inputs could likely crash the monitor entirely. Additionally, model behavior during reinforcement learning training was not monitored at all when the audit began.

OpenAI stated that it paused reinforcement learning training following an incident involving Hugging Face in August. The company expanded monitoring to include this process after the auditors' access period ended. OpenAI confirmed that it has fixed all the weaknesses identified by the researchers. The full report consists of 61 pages and was distributed internally. After redactions, the leadership of METR and Redwood Research received a 38-page version, while a short summary was released to the public. METR said it had no right to disclose the redacted portions on its own.

For future audits, METR expressed a desire for greater public transparency. The organization aims to push for more open reporting on how internal monitoring systems perform. The current evaluation serves as a baseline for future outside reviews of AI safety infrastructure.