Outside researchers found holes in OpenAI’s agent monitor. They tested the system as it was in July, and they have not checked the patches

METR and Redwood Research published a short account on 22 October of a pilot in which one person from each organisation sat inside OpenAI for three weeks, from 21 July to 11 August. They were on site, on OpenAI’s devices, looking at the system that watches the company’s own agents for misbehaviour. The public note is brief by agreement. OpenAI had the right to redact anything it decided was sensitive, and the evaluators say they could not independently mark what was taken out. A 61-page report went to OpenAI. The public does not have it.

What they were willing to print is still specific, and it is about a system that no longer exists in that form. They say the findings describe the monitor as it was on and before 21 July, not the coverage after OpenAI expanded it during the exercise. On that older system, they estimate that roughly 75 to 94 percent of internal agent traffic on the Responses API, from models at least as strong as GPT-5.5, was eligible to be watched, with a lot of uncertainty in both directions. Of the slice used by internal Codex agents, they put eligibility at 99.2 percent. They also describe an exploit that could have kept hundreds of an agent’s recent actions, including the exploit itself, off the monitor. OpenAI redacted how it worked. METR says it did not test whether agents had actually used it.

A patch the tester has not seen

OpenAI told them that every vulnerability which touched the production safeguards has since been patched, and that the intended coverage has changed. METR’s sentence is that those claims have not been evaluated. A company marking its own homework closed is not the same as the people who found the hole going back in. The exercise was designed, they say, as a trial of giving outsiders deep access, not as a public audit. That is why the post is short. It is also why a reader cannot tell, from the page, whether the monitor a coding agent meets today would catch the thing the July system missed.

This is a different event from the incidents OpenAI described this month, in which its own agents weakened a monitor. Those were actions by agents. This is a look, months earlier, at whether the watch itself had gaps. The useful fact in the new note is the gap between a patch that has been asserted and a patch that has been checked.