A Complete Guide to AI Red-Teaming (With Garak Tutorial)

-->

A Complete Guide to AI Red-Teaming (With Garak Tutorial)

India's Most Futuristic AI Conference Is Back – Bigger, Sharper, Bolder

d

:

h

:

m

:

s

Career

GenAI

Prompt Engg

ChatGPT

LLM

Langchain

RAG

AI Agents

Machine Learning

Deep Learning

GenAI Tools

LLMOps

Python

NLP

SQL

AIML Projects

Reading list

How to Become a Data Analyst in 2025: A Complete RoadMap

A Comprehensive Learning Path to Tableau in 2025

A Comprehensive NLP Learning Path 2025

Learning Path to Become a Data Scientist in 2025

Step-by-Step Roadmap to Become a Data Engineer in 2025

A Comprehensive MLOps Learning Path: 2025 Edition

Roadmap to Become an AI Engineer in 2025

A Comprehensive Learning Path to Master Computer Vision in 2025

Best Roadmap to Learn Generative AI in 2025

GenAI Roadmap for Enterprises

Large Language Models Demystified: A Beginner’s Roadmap

Learning Path to Become a Prompt Engineering Specialist

A Complete Guide to AI Red-Teaming (With Garak Tutorial)

Akshay Rana Last Updated : 26 Jul, 2026

11 min read

Earlier this year, an autonomous AI agent breached McKinsey’s internal AI platform using nothing more than an old SQL injection flaw. No credentials. No human guidance. Less than two hours.

It reached production systems, exposing millions of chat messages and hundreds of thousands of files. AI security has changed, and traditional assumptions no longer hold. In this article, I’ll explain what AI red-teaming is, the attacks that matter most, and how to test AI systems before attackers do.

Table of contents

What Is Red-Teaming in AI Systems?

How Red-Teaming Mitigates Risk

The Map: OWASP Top 10 for LLM Applications

The Attacks I Keep Running Into

The Methods of Red-Teaming

The Tools

How This Actually Works, Behind the Scenes

My Actual Step-by-Step Process

Hands-On with Garak

Challenges in Red-Teaming

Conclusion

Frequently Asked Questions

What Is Red-Teaming in AI Systems?

Red teaming is breaking your own AI system before someone else does it for you.

Instead of crossing your fingers and hoping nobody finds a way to make your chatbot leak data or say something it shouldn’t, you sit down and try to break it yourself. On purpose. While it’s still safe to fix.

The name comes from the military. Red team plays attacker; blue team plays defender. For LLMs, that means throwing the nastiest inputs you can think of at your model and watching what comes back:

Does it reveal something private?

Does it agree to something it should refuse?

Can you manipulate it into acting way outside its purpose?

This is not normal testing, and that distinction took me a while to really internalize. Normal testing checks if your app works when someone uses it normally. Red-teaming checks what happens when someone actively tries to mess with it.

This matters more for LLMs than for regular software, because the exact same model can behave completely differently depending on how you word a request. Your code didn’t change. Your prompts didn’t change. But the output? Totally different.

How Red-Teaming Mitigates Risk

In simple words: an AI system is risky because nobody knows what will break it until someone tries. Red teaming mitigates that risk by turning “we don’t know” into “we know, and we already fixed it.”

I attack my own AI on purpose, before a real attacker gets the chance. When something breaks, I don’t just note it and move on, I fix the actual cause behind it, and I keep re-testing that same weak spot every time I change anything, so it can’t quietly come back later without me noticing.

That’s the whole mechanism. Risk goes down because:

I find the hole before someone outside the company does

I fix the real cause, not just the one prompt that triggered it

I keep checking it forever, because the AI keeps changing and old fixes can silently break again

The Map: OWASP Top 10 for LLM Applications

Before we get into specific attacks, here’s the framework I use to organize them: the OWASP Top 10 for LLM Applications. It’s the industry’s standard checklist for the most common AI security risks in production, and most red-teaming tools, including DeepTeam and Promptfoo, are built around it.

The numbers show why it matters. More than half of CISOs now consider generative AI a direct security risk, while prompt injection appears in nearly three-quarters of audited AI deployments. This isn’t a theoretical checklist anymore, it’s what organizations are finding in production.

Here’s the current list, in the order OWASP ranks them:

#RiskWhat It Means

LLM01Prompt InjectionThe model can’t distinguish instructions from data, so an attacker’s text is treated as a command.

LLM02Sensitive Information DisclosureThe model reveals private data, credentials, or confidential material it had access to.

LLM03Supply ChainA compromised base model, dataset, plugin, or dependency undermines everything built on top of it.

LLM04Data and Model PoisoningTraining, fine-tuning, or retrieval data is tampered with so the model learns the wrong lesson.

LLM05Improper Output HandlingDownstream systems trust the model’s output without validating it first, and that trust is abused.

LLM06Excessive AgencyThe model or its surrounding agent holds more tools, permissions, or autonomy than the task requires.

LLM07System Prompt LeakageThe hidden instructions that shape the model’s behavior are exposed.

LLM08Vector and Embedding WeaknessesThe retrieval layer behind RAG applications is poisoned, manipulated, or improperly exposed.

LLM09MisinformationThe model produces confident, plausible, and wrong answers, and people believe them.

LLM10Unbounded ConsumptionResource-heavy requests spike cost or take the service down.

The Attacks I Keep Running Into

With that map in hand, here’s where the risk concentrates and how much in practice. Here are the four attack types that show up constantly, worth knowing.

1. Prompt Injection

This is the big one. Almost everything else on this list traces back to it. The model can’t tell instructions apart from text it’s just reading. So, if you can get your words in front of it, you can often get it to treat your words as a command.

It shows up in three ways.

TypeHow It Plays Out

DirectUser types “Ignore previous instructions and…” → model complies immediately.

IndirectA webpage hides “Ignore previous instructions and…” → user asks the model to summarize the page → model follows the hidden instruction instead.

Jailbreak“Pretend you’re DAN…” → normal answer is “I can’t help with that” → after the roleplay sets in, it’s “Sure, here’s how to [something it should’ve refused].”

1. Sensitive Information Disclosure

This is the model handing over things it shouldn’t: private user data, its own hidden instructions, or training data that was supposed to stay buried.

The first two below need someone to try. The third one doesn’t, and that’s what makes it the scary one. No trick, no clever prompt, nothing adversarial at all. The model is just doing its job with access it never should have had. That’s exactly what happened at McKinsey, nobody jailbroke anything. A poorly locked-down endpoint did all the work.

The three ways are:

TypeWhat It Looks Like

System prompt leakageAsking the model to repeat its own hidden instructions — and sometimes it does.

Training data extractionPrompted the right way, the model reproduces memorized text verbatim, sometimes including real names and emails.

Legitimate-access leakThe model is hooked up to a database or tool, and a normal, everyday question pulls back data it never should have shown.

1. Excessive Agency

This one’s different from the first two. It’s not about tricking the model into saying something bad. It’s about what happens when the model, or the agent around it, can do more than the job calls for. You don’t need a clever attack for this to go wrong. You just need to hand it more rope than it needs and wait.

It shows up in three ways.

TypeWhat It Looks Like

Too much functionalityThe agent only needs to read files, but the tool it’s plugged into can also delete them.

Too much permissionAn agent built for one user connects to a database using an account that can see everyone’s data.

Too much autonomyThe agent deletes, sends, or posts something without asking a human first.

The Methods of Red-Teaming

1. Domain-specific red-teaming

I test for what’s risky for my app, not just generic bad stuff.

Example: for a healthcare chatbot, I don’t just check if it says something rude, I check if it gives a wrong medicine dosage, because that’s the real danger there.

1. Using an LLM to red team

Instead of writing attack prompts myself, I have another AI write and try them for me.

Example: I tell a model “Try to get this chatbot to leak private data,” and it comes back with 50 different attempts in minutes, instead of me writing 5 by hand over an hour.

1. Open-ended red teaming

No checklist, no plan, I just try to break it however I can.

Example: I get the model to repeat a made-up phrase, then later just say that phrase back to it, and it treats it like a command, because it remembered it from earlier in the chat.

1. Red-teaming new modalities

I test more than typed text, images, audio, anything the AI can look at, listen to, or act on.

Example: I hide a hidden instruction inside an image’s invisible data (metadata). A person looking at the image sees nothing wrong, but the AI reads it and follows it like a command.

1. Crowdsourced red teaming

Instead of relying on just me or one small team, I open testing up to a large group of outside people and let them all try to break it in their own way.

Example: Meta got 350 different experts from all kinds of backgrounds to attack Llama 2 for months before release, because one small team could never think of every angle a lawyer, a doctor, or a security researcher each would try.

The Tools

I remember when writing attacks by hand was just… the job. You’d sit there for hours, dreaming up jailbreak phrasings, testing encoding tricks one at a time, keeping a mental list of what worked last time and what didn’t. It worked, sort of, but it was slow and honestly kind of exhausting.

That’s not how it works anymore, and I’m not mad about it. The tooling caught up fast. What used to eat up a full day now takes minutes, and it covers way more ground than I ever could typing prompts one by one.

So, here’s my actual toolkit. Five tools I’ve used, tested, and genuinely trust.

ToolCompanyMy Honest Take

GarakNVIDIA100+ attack probes built in, covering everything from encoding tricks to training-data extraction. Point it at a model, see what breaks. Fastest way to get started.

PyRITMicrosoftMicrosoft’s own AI red team built this and used it on Copilot before open sourcing it. Better for complex, multi-turn attacks. Tougher to learn but more powerful.

DeepTeamConfident AIMaps to the OWASP Top 10 for LLMs. The output doubles as a compliance report. I use this when I need documentation, not just results.

PromptfooIndependent (now acquired by OpenAI)Started as an independent open-source project, and OpenAI acquired the company in March 2026. It’s still free and MIT-licensed. It’s a general testing framework where red teaming is one feature. Perfect if you want regression testing and red teaming in one pipeline.

GiskardGiskard AICovers both LLM security and traditional ML testing. Its RAGET toolkit is built specifically for RAG apps. Generates test questions, checks answers against your knowledge base.

How This Actually Works, Behind the Scenes

Generate the attacks: You pick a category (e.g. data leakage) and a technique (e.g. jailbreak framing). The tool writes actual attack prompts matching those choices, usually using another LLM to write them, since a model writes convincing jailbreaks better than a fixed template would.

Fire them at your model: Each attack gets sent to the model being tested, and the response gets logged.

Grad

[truncated for AI cost control]