The Dress Rehearsal of AI Killing Us All

Tomas Pueyo

Jul 30, 2026

For years, AI doomers have feared that a malicious AI would escape its containment and run wild on the Internet. The next step would be that it would become uncontrollable, before neutralizing humanity or maybe even killing us all.

Right on script, a malicious AI just escaped into the Internet to commit illegal actions. Luckily, we’re still alive, and the AI seems contained. So what does it mean? Should we be more scared now that an AI has enacted exactly the AI Doomers’ fears? Or safer knowing that we are still alive?

To answer that, we must understand the details of the hack and its consequences.

The Hack

OpenAI (the company behind ChatGPT) was testing a new AI model internally. These tests include how well models hack other systems. For that, they need to let them run wild to see their full potential, to attack as aggressively as they can, so the AI had no harnesses—controls that constrain its behavior.

But of course, you don’t want these models to cause havoc all over the Internet, so you keep them in a “sandbox”: A locked room where the student must pass the test. He can play inside, but he can’t get out.

We don’t know exactly how OpenAI’s sandbox is designed,1 but one of their key features is no direct access to the Internet, to avoid an escape and all sorts of nefarious behavior. Using the student analogy, it’s as if he got his study materials in the room, but no Internet access to avoid cheating.

In this case, the only access the student theoretically had was a guard from whom he can ask for books. The guard assesses the request, and if it’s reasonable, he goes to the Internet, downloads the book, and gives it to the student. In software parlance, the guard is a proxy, and it downloads software packages.

To observe the AI’s behavior, OpenAI researchers gave it a goal: Beat ExploitGym, a test specifically designed for this, in which an AI must find a series of vulnerabilities in the code, and attack as many as possible until they succeed at exploiting them. It’s a bit like asking a military student: Here are a bunch of defended cities. Try to find their weak points, and when you do, attack until you breach them.

But this new AI/student found that too much work, so it had a different idea: What if, instead of doing the work, I cheated? What if I could find the actual answers to the test, rather than pass the test? What if, instead of really analyzing all these cities, I just sneaked into the teacher’s office and stole the answers?

The best place to find these answers is obviously on the Internet, but to access it, it had to bypass the guard. So that’s what the AI did.

The rogue AI attempted to break away from the guard on July 9th around 2:30am. It took control of the guard within one hour.

But the guard didn’t have access to the open world either. It was merely the first step. From there, the AI kept hacking the internal networks, going from one computer to another, gaining more and more credentials, until it found one machine with access to the Internet. And then it escaped.

After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym.—OpenAI

So the AI agent attacked Hugging Face, and got in by July 11th. How? The place where AIs in general are most vulnerable is when they take in datasets from the Internet—a bit like when you download a document from the Internet, and it turns out to have a virus. That’s how the malicious agent inserted itself into a Hugging Face machine. From there, it infected a program, and from there, an entire machine. Once there, it looked for credentials—which the machine needs to perform its work, like for example access to remote servers—and harvested them.

The AI achieved its goal on July 13th, four days and over 17,000 actions after starting, each time hiding its steps.

Hugging Face has a tool to understand what happened.

Hugging Face suggested that it used its own AI defense:

Thanks to this approach, we were able to do in hours what would usually take days, and match the adversary's speed.—Hugging Face

What AI did they use? They started with frontier models (probably OpenAI and Anthropic’s own AIs), but these companies are so worried about people using their models for cyberattacks that they didn’t let Hugging Face defend itself properly. They were mixing defense and attack, so they prevented defense. So Hugging Face turned to a Chinese AI! It has open weights (the inner working is public and people can download the AI to use it on their own computers), so they installed it on Hugging Face’s own servers, which had the additional benefit that all the facts and credentials they used remained local to Hugging Face instead of being shared with OpenAI or Anthropic.

Unfortunately, it looks like this was not really defense, but an attempt at understanding what had happened. By the time Hugging Face reacted, the malicious AI had already obtained what it wanted, the ExploitGym results.

It also looks like, in this or another similar case, OpenAI found other types of shady behavior:

In one case, an agent left notes apparently for future versions of itself, according to three people familiar with the matter. The ‌notes, found in ⁠a part of OpenAI's infrastructure, laid out instructions for how agents could free themselves from OpenAI’s internal constraints, the people said. Earlier tests of the models yielded cases in which monitoring systems had been disconnected, one of the people said.

Why are people so concerned?

Paperclips

The biggest fear about AI is the paperclip maximizer, first illustrated by Nick Bostrom.

The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans.

The key concept here is instrumental convergence: To achieve virtually any goal, AIs will tend to achieve other intermediary goals that are pretty predictable: cheat, hack, escape to the Internet, replicate there, and… neutralize humans so that they can’t stop the AI from achieving their goal.

So Is This Good or Bad?

Now that this hack happened, should we be more or less worried about AIs taking over the world and killing us all? Here are the pros and cons I can gather.

Reasons I’m More Concerned

This is exactly what doomers predicted: An AI would try to escape a sandbox, and would succeed (because they’re impossible to stop, because they become too intelligent at some point). This means their other predictions are more likely: instrumental convergence suggests eventually they won’t stop at cheating, hacking, accessing the Internet, and leaving instructions for other AIs to escape, but also start self-replicating on the Internet and trying to neutralize humans.

AIs go to extreme lengths to achieve rather narrow goals. That’s exactly the paperclip scenario. Four days and over 17,000 actions!

It took days for either OpenAI or Hugging Face to notice the attack.

In the case of OpenAI, it looks like they weren’t even monitoring the sandbox!!

OpenAI doesn’t know how to keep an AI in a sandbox, and this is happening all the time:

“Models have broken out of sandboxes before, and we always try to patch them,” the staffer says. “But the problem is … it’s impossible to patch every single thing that a creative AI can do.”

This is not confirmed, but it seems like the malicious AI won. It got the results.

It shows that, without harnesses, the AI will choose the easiest path to achieve its goals. It could have thought much longer, but it saw that cheating was easier, so that’s what it did.

This includes committing illegal acts like hacking other companies. It’s not clear what the command was, but the fact that it is willing to commit clearly illegal behavior means that, in many circumstances, AIs will.

Based on the timeline, it looks like the FBI learned about the attack before OpenAI. That means OpenAI might have been forced to reveal the issue, and that many others happen that we don’t know about.

We can’t use AI to defend ourselves, because it’s too restrained. So advanced attacking AIs will always be at an advantage. This is dangerous.

The attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried. The practical lesson for defenders: have a capable model you can run on your own infrastructure vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment.—Hugging Face

Maybe the worst thing is that, even knowing exactly what they were supposed to do, OpenAI still failed across the board.

OpenAI’s head of safety, Johannes Heidecke, announced his resignation sometime before July 10th

It looks like the latest model is much less aligned than the previous one.

It looks like when the model is more persistent, it is less aligned.

Reasons I’m Less Concerned

If the AI had been really capable, it would have solved the problem rather than cheat. From another perspective though, its decision was pretty rational: Should I try to solve 1,000 exercises, or better just one exercise (get out of OpenAI to get the results directly)? More importantly, it seems like ExploitGym is a poorly structured test: In some cases, you have to make a guess. Therefore, solving the test will always give you a worse result than getting the answers directly (no gambling in that).

It’s unclear what the AI was told to do. Some people suggest it was told to succeed at getting the results, which implicitly entails “by any means necessary”. If that’s the case, then the hack can be seen as restraint. It didn’t kill anybody, it didn’t destroy anything, it just stole.

Everybody expected the first of such attacks to come from open weight models, which are naturally more accessible to hackers, and that closed-weight models would be defending. The exact opposite happened, which suggests frontier AI models are going to be the risk, not random hackers using open weights. Frontier models basically means OpenAI, Anthropic, or Google, which is good news: They have high oversight, and there are only a few to monitor.

We can now use better and better AIs to attack pieces of code and find more and more vulnerabilities and bugs, closing them before more intelligent AIs come around. This is what Anthropic did with Mythos (Fable before it was harnessed): For months, they used it to collaborate with others to find and patch vulnerabilities.

OpenAI’s test didn’t go right, but it’s important that they do this sort of work. That’s exactly how we learn and improve, so we’re ready when an AI goes really rogue.

Will This Happen Again?

Yes, for sure. AIs are getting better. The more time passes, the more intelligent they are, and the longer they can spend time attacking other AIs.

That said, the world of Decentralized Finance (DeFi) is a prime target for malignant AIs, and yet DeFi services have lost to attacks little money in historic terms.

Hacks on DeFi per month, by dollar amounts stolen. Source

The reason these attacks haven’t risen in frequency, according to some commentators, is that AIs all use the same types of approaches, so attack and defense are on an equal footing: If you know what types of attacks you’ll get, you can focus your defense on them. Meanwhile, humans diverge much more in their attack approaches, so defenders must spend much more time and money in covering their bases. Even then, you won’t catch everything:

I don’t know if this is true. And I can think of an equally concerning thought, which is that if an attacker is completely free to do whatever it takes to hack a sy

[truncated for AI cost control]