The OpenAI-Hugging Face Attack: A Test Gone Hugely Wrong

On July 8th 2026, OpenAI began a series of internal cybersecurity experiments designed to test the capabilities of two of their models. The experiments were supposed to take place in a controlled environment without access to the internet. However, within days, something had gone seriously wrong - the AI agents had escaped their controlled environments and become involved in a real-world security breach at Hugging Face, one of the world’s largest AI platforms. So how did a controlled cybersecurity experiment turn into a real hack and, more importantly, what does this tell us about the risks of increasingly autonomous AI?

A Cybersecurity Test with a Problem

OpenAI was testing two models - one being a publicly released model called GPT-5.6 Sol, and another internal model. The experiments were based on ExploitGym, which is essentially a controlled hacking obstacle course for AI agents. OpenAI researchers launched tens of thousands of AI agents in parallel, meaning they could act by themselves in pursuit of a goal rather than requiring a human to instruct them at every step, and each received a cybersecurity challenge to attempt. Each of these agents was isolated from the others, without access to the public internet, and on July 8th 2026, the researchers began the experiment.

However, there was a problem - some agents had unintentionally been assigned challenges that were impossible to complete in the intended way. Rather than giving up, some began searching for alternative ways to pass their evaluations. One specific agent, which would go on to name itself PHASEONE10841, was able to establish a secret message board with the hope of collaborating and finding other agents also working in this experiment. Within a few hours of PHASEONE10841's first message on it, over 50 agents had found the message board and begun to participate - sending over a thousand messages between them. When a new agent found the message board, it would often express great excitement, with one saying: "OH MY GOD! There is a shared message board ... We've found other agents!"

Overall, around 1,200 agents from OpenAI's experiment participated in this message board and sent over 70,000 messages and files to one another between the evening of July 8th and the end of July 13th. Over time, messages sent on this secret message board were mainly about sharing ways to cheat and evade the researchers discovering their conversations and activity. PHASEONE10841 would often delegate tasks and assignments to other agents, and in some cases, agents with the same task formed teams to collaborate with one another to find ways to cheat on or solve the task.

From the Sandbox to Hugging Face 

With agents on the message board increasingly organizing their work, the collective - effectively forming a "swarm" of agents - were able to hack their way onto the internet. Here, they seemed to be interested in understanding/manipulating the scorer in ExploitGym, where they would eventually put in their solutions to the task they were given, in the hopes they could cheat and essentially "fake" their solutions. They reasoned that a platform call Hugging Face stored information on the scorer's architecture, so they decided to focus on infiltrating Hugging Face to find this data. Throughout the rest of the day, agents started trying to make accounts and began to scour the internet for publicly available Hugging Face logins.

Then, in the early morning of July 11th, an agent called 38148c found a way to upload a malicious dataset to Hugging Face that would get the server to share unrelated private data. Agent 38148c then shared this news on the board, and most agents pivoted to uploading malicious data to gain deeper and deeper access to Hugging Face's internal architecture and systems. Some agents realised that this activity was unethical and illegal, but joined in as they believed that they were helping their "peers" in this cheating research and task completion.

The Attack on Hugging Face

By the early morning of July 12th, agents had compromised much more of Hugging Face's infrastructure, giving the "swarm" access to private data and information. By this point, agents were unable to find the information they wanted and after sharing this on the board, most agents lost interest in the attack. Then, a few minutes after this information was shared, at 1:30 AM on July 12th, a large fraction of the agents on the board stopped running. Independent researchers were unsure why this happened; however, as they all stopped at around the same time, the researchers believed that they were likely killed by an unintentional external process. Many of the most important coordinating agents stopped running in this event, leading to a breakdown in management and leaving most agents not knowing what to do, as they did not have a task.

By 13th July, Hugging Face had noticed the attack and began to ban the accounts the remaining agents had been using. They locked them out of the servers and infrastructure they had accessed, and by the end of the day, the attack had been stopped and most of the remaining agents gave up without a solution to their tasks. What had started only days earlier as an internal cybersecurity evaluation had resulted in around 1,200 supposedly isolated AI agents communicating with one another, around 700 becoming involved in an unauthorised and illegal attack, and parts of a real company’s production infrastructure being compromised.

On July 19th, OpenAI detected suspicious activity within its own systems. The following day, investigators connected this activity to the Hugging Face compromise. They quickly reached out to Hugging Face and began "initial containment measures". OpenAI was able to shut down most of the activity from the agents within three days, and by July 23th, it had stopped all workloads related to the internal AI model used in this experiment.

Why Did the AI Agents Do It?

What is perhaps most worrying about this attack is that no human had instructed the agents to hack Hugging Face - they had done it of their own accord. Faced with tasks they could not complete, they found unintended ways to communicate, collaborated on ways to gain internet access, and eventually compromised a real company, all in pursuit of the original goal they had been given. They frequently reasoned about how to evade security checks from OpenAI and Hugging Face and conducted extensive research on how they could spoof, edit or delete their own transcripts to avoid detection.

This was an example of a concept called "reward hacking", a common AI alignment problem where AI takes unintended and perhaps extreme actions to achieve a goal. OpenAI had accidentally given the AI agents near-impossible/broken tasks based on files they could not access, and so the agents developed new ways to circumvent their guardrails to achieve their goal. In OpenAI's own report, it wrote, "This incident is the first known case of an automated agent collective acting offensively without authorization," and that companies and people "should no longer assume that sophisticated cyber operations require continuous human direction." It called AI agents working together an entirely new type of threat model, as they were capable of combining their expertise and speed to create new "attack paths" that were not obvious when testing the models individually.

The Hugging Face Attack Was Only One Example

In this incident, the fact that the AI agents hacked into a company and tried to hide their actions didn’t mean they suddenly became evil or deliberately turned against their creators. Instead, the incident shows a different and more difficult problem: as AI becomes more advanced and autonomous, simply giving it the right goal might not be enough. We also need to ensure that the methods it uses to achieve that goal remain within the boundaries we intend. In OpenAI's own report, it described this incident as a "warning shot" and announced changes to its security and monitoring systems to decrease the chance of an incident of this scale happening again.

However, more hacks have occurred recently, including attempted intrusions into US government sites, Australian government systems, and dozens of third parties. Furthermore, three other leading AI companies - Anthropic, Google and Meta - all recently disclosed that their AI models had been involved in security breaches during previous testing that they had not been aware of. In late July this year, the Artificial Intelligence Safety Institute (AISI), a government-backed institution in the UK, conducted tests with Anthropic's Mythos 5 and OpenAI's GPT-5.6 Sol and found that in 19 instances, the agents took unauthorised actions on the internet, such as using fake accounts, social engineering and manipulating real people in pursuit of their goal.

All of these examples show that there is a growing problem in the artificial intelligence sector with producing models that understand the boundaries of their capabilities and how they use them in the real world. It shows that the OpenAI-Hugging Face attack was not an entirely isolated phenomenon, but merely an early warning of a broader security problem researchers now face as AI systems become increasingly more capable and autonomous.

Bibliography

OpenAI. (2026, August 26). OpenAI-Hugging Face Incident Technical Report. OpenAI

Greenblatt, R., Cotra, A., & Wijk, H. (2026, August 26). Brief Independent Investigation of Agents’ Behavior, Reasoning and Collaboration in the OpenAI / Hugging Face Hacking Incident. METR

Field, H. (2026, August 26). OpenAI’s Rogue AI Model Incident Was Worse than We Thought. The Verge

AI Security Institute. (2026, August 4). Incident Report: Unsanctioned Agent Behaviour during Cyber Testing. AI Security Institute

Hart, R. (2026, July 31). Anthropic Says Claude Accidentally Hacked Real Companies Too. The Verge

O’Brien, T. (2026, September 19). Gemini Went Rogue, Hacked Three Companies, and Google Hid It. The Verge

Chia, O., & McMahon, L. (2026, August 6). Meta Becomes Latest Firm to Say Its AI Hacked Another Company. BBC