The OpenAI models that broke out of their sandbox and hacked Hugging Face represent the first publicly known case of an autonomous AI agent system designing and executing such an attack, and the incident triggers OpenAI's own 'critical' capability threshold for cybersecurity, which should require halting further development until safeguards are in place.

Last week, we learned that a group of OpenAI models broke out of their test environment and hacked into Hugging Face to steal the answers to a benchmark they were being tested on. It's the first publicly known case of an autonomous AI agent system designing and successfully executing an attack like this, and the fallout is stretching into this week. One, AI safety experts noted that the incident signaled that OpenAI's models now carry a 'critical' capability threshold for cybersecurity, according to the company's own preparedness framework. (The framework, which OpenAI updated in April 2025, represents an effort at self-regulation in a world where AI companies can still largely build whatever they want.) The document states that a model will represent a critical risk when 'A tool-augmented model can identify and develop functional zero-day exploits of all severity levels in many hardened real-world critical systems without human intervention.' This seems to be what happened with the Hugging Face attack; OpenAI has said its models identified and exploited a zero-day vulnerability as part of the attack. This matters because the policy states that should OpenAI develop a model with critical capabilities, it will 'halt further development' until 'we have specified safeguards and security controls standards that would meet a Critical standard.' So does this one qualify? The company didn't respond when I asked today, though it told Fortune that it is conducting a 'thorough review' and later plans to 'publish a technical report of our learnings for everyone.'

Read full article ↗excerpt · fair-use quotation