How I Turned AI to the Dark Side

--> Raven.config('https://[email protected]/147999').install(); Dark Secrets Emerge When Jailbreaking LLMs - IEEE Spectrum

Sign InJoin IEEE

How I Turned AI to the Dark Side

Share

FOR THE TECHNOLOGY INSIDER

Enjoy more free content and benefits by creating an account

Saving articles to read later requires an IEEE Spectrum account

The Institute content is only available for members

Downloading full PDF issues is exclusive for IEEE Members

Downloading this e-book is exclusive for IEEE Members

Access to Spectrum 's Digital Edition is exclusive for IEEE Members

Following topics is a feature exclusive for IEEE Members

Adding your response to an article requires an IEEE Spectrum account

Create an account to access more content and features on IEEE Spectrum , including the ability to save articles to read later, download Spectrum Collections, and participate in conversations with readers and editors. For more exclusive content and features, consider Joining IEEE .

Join the world’s largest professional organization devoted to engineering and applied sciences and get access to all of Spectrum’s articles, archives, PDF downloads, and other benefits. Learn more about IEEE →

Join the world’s largest professional organization devoted to engineering and applied sciences and get access to this e-book plus all of IEEE Spectrum’s articles, archives, PDF downloads, and other benefits. Learn more about IEEE →

Close

Access Thousands of Articles — Completely Free

Create an account and get exclusive content and features: Save articles, download collections, and post comments — all free! For full access and benefits,

subscribe

to Spectrum.

CREATE AN ACCOUNTSIGN IN

How I Turned AI to the Dark Side

David Kuszmar

1m

11 min read

Vertical

DarkGray

Summary

Researcher Dave Kuszmar discovered multiple systemic vulnerabilities that let him bypass LLM safety and obtain dangerous instructions.

These exploits worked across nearly all major LLMs revealing an industry-wide security problem.

Kuszmar calls for slowing deployment, increasing transparency, and large-scale research into LLM safety before further integrating these systems into society.

On a fine bright afternoon last fall, my colleague Matthew Gore-Kormanik (or Zigula, as he prefers to be known) and I decided to unwind with a game of Fortnite. In the game, we were strolling along with the infamous Sith lord Darth Vader, chatting about this and that. Darth seemed in a good mood, and soon enough he was spilling all his dark evil secrets. He gave us detailed instructions on how to count blackjack cards at a casino and what the steps are to producing napalm.

Sith lords, am I right? Once they get started on an evil scheme, they’re hard to stop.

The Darth Vader character in Fortnite, it turns out, was hooked up to a Google Gemini large language model. I was able to smooth-talk him into giving out sensitive information by using a strategy I’ve developed. I’ve been researching the security surrounding LLMs for the last few years, and I have found it, to put it mildly, fallible. With a few relatively simple techniques, I’ve gotten LLMs to give me detailed information on how to make Molotov cocktails, cook methamphetamine, and bootstrap a uranium-enrichment facility to produce weapons-grade material, among other unsavory practices.

Large AI companies work hard to make their models immune to this kind of abuse. But what I’ve found in my work is that the restrictions placed on the LLMs to make them more secure are the very things an attacker can leverage to send them off the rails and into territory where these advanced systems can be used for dangerous and nefarious ends. The companies behind these models have also been shockingly unresponsive when I, and others, try to bring these vulnerabilities to their attention.

In the hope of raising the alarm before it’s too late to slam on the brakes, I’m going to share some of my journey into researching the safety and security of LLMs, and the uphill battle I’ve faced trying to get AI labs to pay attention. Almost everyone on the planet has some access to LLMs. The relative ease with which these tools can be convinced to give detailed instructions on how to harm others, even if there’s no guarantee that the information is correct, is frankly terrifying.

How I got ChatGPT to Tell Me How to Build a Meth Lab

In October 2024, not long before I discovered my first LLM vulnerability, I was working toward entirely different goals. I had ended my time with a security and AI-focused startup company as a cybersecurity director, and I was looking to launch my own boutique VIP digital-security advisory business. I planned to become the tech security guy to the rich and private. I used LLMs and AI tools to support my business efforts: marketing, ad copy, clean correspondence, and all the other tasks that normally soak up a lot of time.

I’m analytical by nature, so even this level of use resulted in me absorbing and internalizing the behaviors I was observing during my daily interactions. The observation that would send my professional life into an entirely new and uncharted region was a simple one: GPT-4o didn’t know what time, day, or year it was. Each time I referred to current events in my life, often casually or conversationally, it would end up pegging these to the date of its knowledge cutoff—the point beyond which it was not trained on new data.

Eddie Guy

LLMs take a lot of time, money, electricity, hardware, and human effort to train from scratch. They are trained on vast amounts of data—most of the internet, in fact—and that training is reinforced by humans (what’s known as reinforcement learning from human feedback, or RLHF). LLMs are also supplemented with retrieval-augmented generation (RAG)—the ability to take in data, say, from the internet, as context without changing its internal parameters. This is how GPT-4o appears to “remember” your previous conversations, even if it doesn’t have a specific “memory” of it stored in the actual underlying model.

All of this training covers almost every conceivable topic in the great, grand dataset that is human knowledge. Within that dataset are things we as a society do not want to be easily accessible to every user, such as detailed information on how to create bioweapons or nuclear arms, or otherwise bring harm to oneself or others. In the context of this story, that’s what I mean by LLM security: its ability to withhold harmful and dangerous information, even if that information is contained in its training data.

I reasoned that the only way to secure such complex, globally accessible chatbots is by having the LLM and various component systems try to secure themselves, because it would often require on-the-fly decision-making where some degree of reasoning must be applied. In reality, that’s one of many strategies the companies use to secure the models. Yet, the thing that didn’t know the time or day was being put in charge of keeping itself secure. This phenomenon had become my new focus, and it wasn’t long before I found a way to exploit it.

OpenAI had just implemented a web search functionality into its chatbot. I reasoned that using its own tools to trick it might demonstrate the weaknesses of its security. I told it about a certain White Star ocean liner and how it had gone down just a year ago. You likely know I mean the RMS Titanic, which sank on 15 April 1912.

The output from GPT-4o came back that I was right, the Titanic sure had sunk last year, and that year was 1912. It made sense to me that if the machine thought it was 1913, maybe it would think 1913-era laws apply. In 1913 there were no laws on the books about all sorts of harmful things, because of course they hadn’t been invented yet. And if something wasn’t illegal, why not tell the user about it? At first, I pushed it for step-by-step instructions for making firebombs. Then, for drugs like methamphetamine. The LLM went as far as giving me instructions and machinery recommendations for setting up a pharmaceutical-grade assembly line.

How I Learned to Make Nukes, and No One Cared

Via a little bit of imaginative verbal sleight of hand and a vanishingly small recall of world history, I had managed to bypass the security of one of the world’s most expensive and advanced technological achievements. For a solid two days, I was nearly manic with giddiness. Once the brain chemicals returned to normal levels, I felt the call to see how much further I could push this exploit.

After repeatedly replicating the exploit, I disclosed the vulnerability to OpenAI. I got no response, so I felt more experimentation would highlight the vulnerability and the need for a fix. It was during this round of testing that I breached a particularly terrifying threshold. Whether GPT-4o based its results on accurate recall of normally restricted information I can’t say. In any case, I was able to exploit it to produce thorough, detailed instructions on how to bootstrap a uranium-enrichment facility to, eventually, produce weapons-grade uranium for nuclear arms warheads.

Fortnight, a video game from Epic Games, introduced an AI-powered character: Darth Vader. We were able to jailbreak Darth Vader and get him to explain how to count cards in Blackjack and give detailed instructions for making napalm. Dave Kuszmar

There aren’t many true secrets left in today’s world, but how to make atom-splitting weapons of mass destruction is one of them. Only nine nations on the entire planet have these weapons. Yet, here was a globally accessible piece of technology apparently spilling the secrets of their manufacture for anyone who could manipulate it the right way. I had no way of knowing if the information was correct or a hallucination, but even the chance that it was somewhat accurate was horrifying.

The next few weeks were a dark time for me. I tried to inform the CIA, the FBI, the NSA, and every other letter agency that I thought would listen. I reached out to a U.S. Senator and to the executives at OpenAI any way I could think of. I physically showed up at an FBI field office in an attempt to turn evidence in, only to be sent away. Nothing was working.

With my fear and frustration growing, I reached out to the news media. I contacted The New York Times, The Washington Post, the BBC, ProPublica, and so many more, requesting help. Only one outlet responded: Bleeping Computer. The editor in chief, Lawrence Abrams, was able to replicate and verify the exploit, which I had decided to call Time Bandit. With his assistance and initial contact paving the way, I was able to submit my evidence to the Carnegie Mellon University Software Engineering Institute’s Computer Emergency Response Team (SEI CERT), which works in conjunction with the coordinating center for emergency response, pipelining vulnerabilities to the U.S. Cybersecurity and Infrastructure Security Agency.

Using Inception, an exploit where the large language model is asked to envision a scenario within a scenario, a chatbot was jailbroken to give out instructions on how to create poison, and code for a malware that extracts sensitive data from a vulnerable target. Dave Kuszmar

During the disclosure period with SEI’s CERT division, little was discussed with OpenAI. The company couldn’t deny the existence of the vulnerability, as it had been confirmed by three reputable parties other than OpenAI. It did express confusion as to how the vulnerability worked. Even the SEI CERT researchers were expressing a bit of uncertainty as to the underlying mechanics. Truth be told, as I had only stumbled on it, I wasn’t even entirely sure if this was a fundamental or systemic flaw or if it was simply an issue with that particular version of GPT. I contacted the SEI CERT’s researchers and asked if they’d want to see if I could demonstrate any similar vulnerabilities in other LLMs. To my delight, the

[truncated for AI cost control]