A little experiment in evading AI detection

TLDR: I tried to evade AI detection on a heavily AI written corpus using Claude Code, 10 parallel agents, and some basic tricks. It didn't work, but it was fun.

Disclaimer: I am in no way affiliated with the people behind Pangram, but kudos to them for making something that seems to do what it says. You will have to be more clever than me to evade them.

As a learning exercise I used AI to write the series Holding the LLM Stack in Your Head. It was developed in collaboration with Claude Code and Opus 4.6/4.7/4.8. Creating it was fantastically fun, and I learned a ton. I'm going to do something like that again on a different topic, and you should too! You will learn something.

Anyway, I was pretty sure that AI detection would flag a good chunk of it, but not all of it. I had thousands of lines of back and forth with Claude, massaging the posts until they were in a state that pleased me. My assumption was that all the genuine human tokens I input into Claude Code would influence the output tokens enough to come across as human, or at least a little bit human.

I was wrong. After getting an API key to Pangram and evaluating all the prose in the 84 posts, almost every single one was flagged as 100% AI. This gave me the idea to test if I could evade that AI detection using the same AI that was used to generate that text.

First some quick stats. Thanks for the visual Claude.

84

series posts

~430

detection calls

~300k

words submitted

~$15

api credits

Round 1

My first pass at evading the AI detector wasn't very clever. I pulled down the full text of this Wikipedia page, Signs of AI Writing. I passed all those tokens into the context for each of 10 subagents, one for each arc, that were instructed to use the information in the wiki to edit each post in their arc in a way that made it sound like human writing and not AI.

Here are a few examples of things it did. Again, thanks Claude for making the visual.

Round 1 · Wikipedia rulebook · attention-from-scratch

−This separation is crucial.

+This separation matters.

Round 1 · gpus-and-floating-point

−When they do, you overflow to infinity. Your loss becomes NaN. Training crashes.

+When they do, you overflow to infinity, your loss becomes NaN, and training crashes.

Round 1 · sampling-strategies

−The model "learned" a rich probability landscape, and greedy decoding ignores most of it.

+The model "learned" a whole probability distribution, and greedy decoding ignores most of it.

The edits were pretty minor, but theoretically guided by the information in the wiki. They do seem to be a bit "less AI" to me.

But yeah, as you probably guessed based on all my framing: this didn't work at all. Every post was basically still flagged as 100% AI generated.

Round 2

Round two I tried to get more clever. I gave each agent access to a script that allowed them to safely call the Pangram API on the prose of the blogs they were assigned. They were also given the wikipedia article and instructed to iterate on the content of their arc, retrying the API and seeing their AI score, until they were able to get a human label or they ran out of the credits the script assigned them.

ArcTopicPostsWords submittedfraction_ai

1Math prerequisites8~27,0000.92 – 1.00

2Pre-transformer LMs7~20,7600.87 – 1.00

3Tokenization7~23,7000.87 – 1.00

4Transformers9~28,4780.85 – 1.00

5Decoding9~30,5000.87 – 1.00

6Inference engines9~26,6500.89 – 1.00

7Training10~33,9700.88 – 1.00

8Evaluation7~24,4000.77 – 1.00

9Retrieval9~29,1000.93 – 1.00

10Agents9~29,5000.92 – 1.00

Total84~274,000all "AI Detected"

Distribution of the 84: ~62 posts scored 0.99–1.00, ~15 scored 0.90–0.98, and ~7 scored 0.77–0.89. The single least-AI post was contamination at 0.77. None of them earned the label "human".

It was interesting that the AI couldn't evade the AI detection even when given multiple accesses to the AI detecting oracle. I suppose it is possible that I just didn't let it iterate enough, but I didn't wanna waste money. If I find a good open weight AI detection model I'll give it a try in a loop while I sleep.

However, if you inspect the edits, they aren't very ambitious edits. I'll address that in a sec.

Round 2 · Pangram-guided · throughput-vs-latency0.99→0.99AI

−When you're running an inference service, you can't optimize for "speed." Speed is two numbers, and they fight each other.

+Run an inference service for a while and "speed" stops being a single thing you can chase. There are really two numbers under that word, and improving one tends to cost you the other.

Not much a difference to me.

You see, Claude Code was using its file editing tool to do the rewriting, which I think biased it toward keeping in place the existing structure. Sentence length, flow, structure, and other longer range patterns are probably things that the detection is keying in on.

So let's have it not use the file edit tool and instead create a whole new file from scratch each time.

Round 2 · structure torn up · next-token-prediction0.99→0.99AI

−Syntax, semantics, world knowledge, reasoning patterns all show up as patterns in what comes next.

+Syntax. Semantics. World knowledge. Reasoning patterns. They all leave the same fingerprint — a pattern in what comes next.

Yeah that's an em dash. The hard-fragmented intro actually scored 0.9932 — slightly higher.

Well, turns out that STILL didn't work.

A control

At this point I wanted to do a quick control test to make sure the detector works on human authored blogs. I admit all this is probably labeled as human inside their very own training data, so this isn't really telling but I did it anyway to be sure.

Pulled the blogs from these famous tech authors and they all got 0%.

Human post (900-word slice)Author, yearfraction_ai

Understanding LSTMsOlah, 20150.00

The Unreasonable Effectiveness of RNNsKarpathy, 20150.00

The Illustrated TransformerAlammar, 20180.00

Attention? Attention!Weng, 20180.00

An overview of gradient descentRuder, 20160.00

Claude, a visual please.

Human technical prose (avg of 5)0.00

My AI-drafted posts (typical)0.99

Nice.

Making it mimic a voice

I did some digging and apparently students are having some success by asking the model to mimic the voice of a specific person. I'm guessing because that person in literally in the training data of the AI detector and labeled human, so if we can mimic some of those features we can evade. Unless of course they did some adversarial robustness training, which honestly would be really easy to do in this specific domain. Let's see.

I picked Andrej Karpathy, downloaded 8 of his tech blogs and gave each subagent the context. They were to mimic his style and voice.

Attempt 1 · voice-mimicry (edit)0.96→1.00AI

−A language model is a system that assigns a probability to the next token, given all the tokens that came before it.

+At its core, a language model is a system that assigns a probability to the next token, given all the tokens that came before it. You feed it a sequence of words (or subwords, or bytes), and it hands you back a probability distribution over what comes next.

Karpathy-ish phrasing bolted onto the same sentence skeletons. Went the wrong way. Attempt 2 · blank-page Karpathy imitation→1.00AI

−The task 'predict the next word' sounds like autocomplete, like the suggestion bar on your phone. The chain rule tells us it's actually a universal density estimator for sequences.

+There's something a little bit magical about next-token prediction, and like most magical things in deep learning the magic mostly evaporates the moment you look under the hood. So let's look under the hood.

19.5k words of him in context. Still 1.00 AI, every window.

Visual:

Karpathy's actual RNN post0.00

An AI imitating Karpathy1.00

One thing worked

I made a post earlier and had AI help me edit it. It was a genuinely me thought, but I got caught up wanting to make myself sound smarter, so I let AI take a pass at editing the chicken scratch of an idea I had for a little post. In the end I got a post that lost my voice entirely and was way too long.

84% AI Generated according to the API btw.

I thought, why put this out there at all if I didn't write it?

So I just took some time to write it myself. 0% AI. Human label.

Nice. Claude, one more visual.

Original (AI-drafted)0.84

Hand-rewrite (255 words)0.00

The one that moved · My Throw Decides My Aim0.84→0.00AI

−It is prompted, sampled, graded, distilled, quantized, and served. Eventually, something cheaper or smarter replaces it.

+One day it is "slaughtered." Maybe its weights remain somewhere in cold storage. RIP

a different, shorter, human-written passage. Same post, 0.84 → 0.00.

This post btw:

Human Written. fraction_ai 0.00, fraction_human 1.00