AI Is the Ultimate Leaky Abstraction

An abstraction is a promise that you won’t have to look underneath. Generated answers make that promise too, right up until the moment they break it, and the moment they break it they hand you back a bill for exactly the understanding they let you skip.

I left a promissory note at the end of the last post. I’d spent the whole thing arguing that every abstraction is really a contract (here are the inputs, here’s the behavior you’ll get back) and that it loses nothing only when the part it hides produces no effect outside that contract, that almost nothing real clears that bar because the hidden part nearly always leaks something, some side effect, some more or less than the behavior you were promised, and then I pointed at the one abstraction eating the industry right now and said it deserved more than a paragraph. This is the paragraph, grown up. If you want the machinery underneath the claim, that post is where it lives; here I just want to follow it to the end of the line.

You don’t need to write code to know this feeling; you just need to have leaned on something that mostly works. Think about the auto mode on a camera. It’s a lovely abstraction over a genuinely hard problem, the three-way balance of aperture, shutter speed, and ISO that decides how a photograph comes out. Point it at a sunny street and it nails the shot every time, and you get to never learn what those three dials actually do. Then you try to catch your kid blowing out birthday candles in a dim room, or a friend backlit against a bright window, and the camera’s little exposure model guesses wrong the way it was always eventually going to. The candles smear into blur, or your friend goes to a black silhouette in front of a perfectly exposed window. Nothing is broken. The camera did exactly what it was built to do. But the only way to rescue the shot is to drop into manual and actually understand the exposure triangle the auto mode spent years letting you ignore, and if you never built that understanding, the moment is just gone. The abstraction was free right up until the instant it wasn’t, and the price it charged was the exact knowledge it had been saving you from. Same story in the driveway, by the way: the car starts every morning and you never think about the battery, right up until the cold snap when it doesn’t, and the thing that let you not think about it is suddenly the only thing worth knowing.

This is the whole pattern, and it’s everywhere once you can see it. Computing is just where somebody finally bothered to write it down. Start where the idea started. In 2002 Joel Spolsky wrote down a “law” every working engineer already knew in their bones but had never quite named. (It’s a “law” the way Murphy’s is, an observation that keeps being right, not the kind physics or a statute can hold you to.) He called it the Law of Leaky Abstractions, and it has one clause: all non-trivial abstractions, to some degree, leak. TCP promises you a reliable stream of bytes and hides the snarl of dropped packets and retransmissions and three-way handshakes underneath. The promise holds almost all the time. Then a backhoe somewhere severs a fiber, a router upstream starts flapping, and the reliable stream stops being reliable in a way no amount of staring at your own code will explain. You open a packet capture, watch the retransmissions stack up, and there you are, debugging a layer you were assured you’d never have to see. The abstraction saved you time while it worked. It never saved you the obligation to understand what it was standing on.

That last bit is the part people forget, and it’s the heart of Spolsky’s “law” rather than a footnote to it. An abstraction doesn’t erase the complexity beneath it. It hides the complexity and changes when you pay for it. While the illusion holds you’re fast, productive, blissfully ignorant of the planner choosing your query path or the allocator deciding where your objects live. The bill comes due at the leak, and the currency the leak demands is comprehension: a working mental model of the substrate, assembled under pressure, usually at the worst possible hour. (This is the same bill I called debt a couple of posts back. Same loan, different collector.) Spolsky’s quiet warning was that as our tools climbed higher, the people maintaining the stack had to know more, not less, because every new floor was one more height the leak could fall from.1

The interface over reasoning you never see

Now set a generative model on top of that stack, and notice it’s a different kind of layer entirely. When you ask a model to write a function, summarize a contract, or propose an architecture, what comes back is an abstraction over a process of reasoning the model never shows you. The prompt goes in. Something happens inside a distribution of hundreds of billions of parameters that nobody can hold in their head. Fluent prose comes out, or code that compiles, or a confident three-paragraph summary that reads like the work of someone who understood the material. The surface has the texture of competence. It reaches for the right nouns, the right libraries, the right register. And here’s the move worth sitting with: the output is an interface, and like every interface it’s a promise that you don’t need to know what happened underneath in order to act on it.

For most of computing history the promise leaked too, of course. If code matched intent perfectly we’d have far fewer bugs, and nobody would have had to invent return-to-libc to climb around a stack canary, or chain gadgets to chip away at ASLR. Intent and reality drift apart in odd ways, and a lot of people make a living in that gap. What made those leaks survivable was inspectability: the substrate underneath was always there to climb down into. This is the difference that matters, and it’s easy to miss because the word “abstraction” papers right over it. When a traditional abstraction leaks, you can descend. The compiler’s output disassembles. The query planner explains its choices with one command. The network stack sniffs packet by packet. The substrate under every classical abstraction is hard, but it’s readable, deterministic, and there: someone forced to can climb down through the layers and reconstruct, step by step, exactly why the system did what it did. Spolsky’s developer, stranded by a leaky TCP connection, had somewhere to go. The leak hurt, but it was legible, and legibility is the thing that makes a leak survivable.

A generated answer withdraws that bargain without telling you. The model’s interior isn’t intricate-but-deterministic engineering; it’s a probabilistic latent space, and the path from any one parameter to any one token can’t be traced by the person leaning on it. There are no intermediate stages to inspect, because the reasoning was never broken into stages. There’s no log of the deliberation, because there was no deliberation in the auditable sense, just a sample drawn from a distribution. So when this abstraction leaks you’re not in Spolsky’s situation at all. You can’t descend into the model the way he descended into TCP. The leak surfaces inside an artifact you didn’t write, produced by reasoning you can’t replay, and the only road back to solid ground is to reconstruct, after the fact, the mental model the model itself never had. The time you saved during generation gets repaid, with interest, during diagnosis. And the interest compounds hardest in exactly the places the stakes are highest, because those are the places the leak is quietest.

The leak you don’t see coming

Let me make this concrete with the kind of failure that doesn’t announce itself. Say you ask a model for a small piece of concurrent code: a counter shared across worker threads, bumped as jobs finish, so a dashboard can show progress. The model hands back something clean. It reads the current value, adds one, writes it back. It’s commented. It compiles. You run it with a handful of jobs and the count is right every time. You write a unit test with two workers and it passes. The demo works. By every check you’re in the habit of running, it looks like understood code.

It’s ninety-five percent correct, and the missing five percent is a race condition.2 Read-modify-write on a shared counter isn’t atomic. Two threads can read the same value, both add one, both write back the same result, and now you’ve counted one job instead of two. Under the load of your test, two workers and a few jobs, the threads almost never collide at the exact wrong instant, so the bug is invisible. Under production load, hundreds of workers hammering the counter thousands of times a second, the collision stops being rare and becomes constant, and your progress dashboard drifts quietly below the truth. Nothing crashes. No exception fires. No test goes red. The system is locally correct and globally wrong, and it stays that way until somebody notices the numbers don’t add up and goes hunting for a cause the generated code gives them no help in finding.

This is the leak in its purest form. The abstraction didn’t fail loudly the way a hallucinated API or a syntax error fails (both of which at least slam into reality right away and leave you the gift of an error message). It failed silently, in the gap between appearance and behavior, under a condition the prompt never specified and the dev environment never exercised. Recovering from it needs the exact thing the generation let you skip: a working feel for memory ordering, atomicity, and why time isn’t as simple as the syntax makes it look. If you fought your way to that understanding once, debugging a deadlock at two in the morning, you’ve got it, and you’ll smell the bug before you finish reading the function. If you’ve only ever accepted generated concurrency code that worked in the demo, the model has handed you a fault whose root cause lives a full layer below anything you were ever asked to learn. The same fluency that produced the bug is no help at all in finding it.

The confident summary leaks the same way, in a register that travels further. Ask a model to compress a long technical document, a vendor contract, a post-incident review, and you’ll get back something smooth, well-organized, authoritative. Almost everything in it is right. But summarization is compression, and compression is a string of decisions about what’s safe to throw away, and the model makes those decisions by statistical salience, not by knowing which clause is load-bearing. The one caveat that mattered, the carve-out that voids the warranty under exactly your conditions, the single sentence in the post-mortem that breaks the tidy narrative, is precisely the low-frequency, high-consequence detail a fluency-optimized system is most likely to smooth away. The summary doesn’t look incomplete. Incompleteness has no texture. It reads as whole, you act on it as whole, and the gap only shows up when the dropped caveat comes back wearing the costume of a surprise.

When the abstraction is knowledge

George Box gave statisticians a line they’ve repeated for fifty years: all models are wrong, but some are useful.3 A model is a deliberate simplification, valuable exactly because it leaves things out, and dangerous exactly when its users forget what got left out. A generated answer is a model in this sense, a compressed, lossy stand-in for a reasoning process and a body of fact, and it’s useful for the same reason it’s wrong. It discards. McLuhan4 would push it one step further. The medium is the message, he said, meaning the form of a communication system reshapes the people inside it more than any particular content it carries. The form of the generated answer is the answer-with-its-derivation-removed, and a civilization that meets its knowledge in that form gets slowly trained to treat the surface as the substance, to take the conclusion and lose the habit of asking what it stands on.

Which is where the l

[truncated for AI cost control]