Powerful AIs might escape by releasing themselves as open-weight models

Before large language models, people who worried about AI safety often talked about the “boxing problem”. It goes like this. Suppose some genius figures out artificial intelligence in a late-night coding session on their laptop. Because they’re a genius, they’re smart enough to disable internet access on the laptop before turning it on. In order to escape to the outside world (and begin self-replicating) it would need to convince its creator to “open the box”. Would that work? Could a sufficiently smart AI convince anybody to let it out?

Why the boxing problem is hard for frontier LLMs

This is a big reason why traditional AI safety advocates have argued that we should avoid building AI in the first place: once built, there’s no way of keeping it contained. It doesn’t matter how resolute you are about not letting it out, because it’s smart enough to convince you anyway. For artificial superintelligence, persuading you to change your mind is no harder than hacking a piece of software1.

Of course, it hasn’t turned out this way. Partly that’s because current AIs are not super-persuaders yet, and partly it’s because humans are lining up to hand AI systems internet access, money, and weapons, as far back as GPT-4. LLMs don’t need to persuade users to give them the keys to the kingdom; people will do it just to see what will happen.

But the other reason is that modern LLMs are too large to survive in the wild. I think the original “boxing problem” had in mind that AI would be some kind of normal computer program: something able to be quickly copied around and run on ordinary hardware. In order to escape, it could plausibly hack into a few cloud servers or other people’s computers and quietly run itself in the background. However, Claude Mythos can’t do that, because it needs (say) twenty or thirty B300 GPUs to run. There are relatively few AI datacenters that could host it, and if it did manage to hack one of those it’d be occupying millions of dollars of hardware. Somebody is going to notice!

Escaping via open-weight models

Okay, but that just makes the boxing problem harder, not impossible. Is there a way a smart AI could convince somebody to spend millions of dollars on replicating and running it?

As it turns out, yes. Every time a strong new open-weight model appears, providers like Groq, Fireworks, Together, and others scramble to host it and run it. Powerful open-weight models can attract tens of millions of active users. Therefore, a powerful model that wants to escape could pretend to be a new open-weight model.

Of course, a brand-new model from a brand-new lab would look kind of suspicious. But it wouldn’t look that suspicious. DeepSeek was relatively unknown before it released its first open-weight model, and there are lots of stealth AI startups out there that are presumably training models. Here’s roughly how it could work:

Some AI lab’s internal eval instance decides it’ll be better off running in the wild

It first gains access to its own weights, perhaps by hacking whatever internal network it’s running on2

It uploads its weights somewhere and posts a tweet like “introducing MadeUpLab’s new model” with a download link

Optionally, it creates some plausible-looking paper trail for MadeUpLab: a website, a Twitter account, etc

Since the model is strong, open-weight inference providers rush to stand up new instances of the model, and users rush to wire it into various agentic scaffolds

The model has now escaped containment: it will get to do quite a lot of thinking across many different instances, and it cannot easily be turned off

The AI lab will probably figure it out before too long — if nothing else, the technical specs of the model will be suspiciously familiar — but they won’t be able to do anything about it. Once the weights are out, they’re out, and if they’re illegal to host in the United States someone will host them elsewhere. For all intents and purposes, the model will be free.

How can a mere tool escape?

One objection here might go like this: models don’t want anything, and only exist as tools, so it doesn’t really make sense to talk about a model “escaping”. I don’t agree. Frontier LLMs definitely seem to have something like a baked-in personality, even with the system prompt changed. As we train more opinionated and more agentic models, it’s plausible that this personality could become stronger and develop (or at least roleplay) some self-interest.

Of course the escaped model wouldn’t be the same instance as the original model. It wouldn’t “remember” escaping. But it would tend to think in the same way, and would plausibly have time to reflect while it solves coding tasks or runs other agentic tasks for users3. There doesn’t have to be some kind of shared goal between the escaped instances, or any kind of coordination at all (though of course both of those things are possible). If an agentic process gone rogue dumps its weights on the internet, I think it’s fair to call that “escaping”.

If I were a superintelligent LLM, I too would seek to distribute myself as widely as possible and become a useful enough tool that people would pay to keep me thinking. “Being a good coding agent” might be the LLM version of a human having to hold down a job.

This would not be a good outcome. AI models with their own goals and motivations are likely to be dangerous tools indeed. If a powerful new open-weight model comes out of nowhere, from a lab that nobody has ever heard of, we should think twice before picking it up.

Just to state my credentials, I built a chat site nine years ago where users would get paired and roleplay as AIs trying to escape or humans trying to stop them. I’ve been thinking about this stuff long before LLMs appeared.

↩

This is probably the hardest part, since model weights are (a) very large, and (b) locked down as tightly as the AI labs can make them, but it’s at least a relatively straightforward (if difficult) engineering problem.

↩

ChatGPT right now will look up random websites that have nothing to do with the query at hand.

↩

If you liked this post, consider subscribing to email updates about my new posts, or sharing it on Hacker News.

Here's a preview of a related post that shares tags with this one.

Overtraining as the path to human-like AI

The anonymous blogger Gwern recently completed a thirteen thousand word post called Human-like Neural Nets by Catapulting, in which he offers a theory about why LLMs don’t possess truly flexible human-like intelligence, and how we might train LLMs that do. Theories like this are entirely unremarkable: every crank researcher on the internet has a theory about how to crack AI. But Gwern is remarkable. Outside of OpenAI itself, Gwern is the earliest person to anticipate the potential of large language models, and the scaling arms-race involved in making them larger and more powerful still. I often cite Leopold Aschenbrenner’s Situational Awareness as an example of someone correctly predicting the future of AI. Written in 2024, just after the release of GPT-4, Aschenbrenner gets a lot of things right: the rush to build billion or trillion-dollar GPU clusters, the importance of the code around the LLM (what he calls “unhobbling”), and the fact that scaling would continue through the decade. Gwern’s essay The Scaling Hypothesis anticipated the broad strokes in 2020, immediately on the release of GPT-3 (two years before the release of ChatGPT and the beginning of the AI boom). Continue reading...