Nothing Is Easy When You're an LLM: The Flat Latency Problem

Dr. Ryan Rad

Jun 05, 2026

When you walk to the kitchen, your brain barely registers it. When you solve a differential equation, it burns. Humans spend energy proportional to the difficulty of the task.

LLMs don’t.

Ask a model where the capital of France is. Ask it to explain recent advances in quantum computing. To the model, both answers cost exactly the same — one full forward pass per token, every time, no exceptions. Predicting the word “Paris” and the word “Qubit” are computationally identical events.

That is not an inefficiency. That is the architecture. And it’s costing you more than you think as it’s integral into every single interaction you have with LLMs.

The Uniform Compute Problem

Every token your model generates receives identical compute — one full forward pass through every layer, every attention head, every weight. An obvious continuation produces a sharply peaked output distribution, one candidate dominates, the answer is never in doubt. A critical reasoning token, that determines the entire downstream branch of your pipeline, produces a near-uniform one, several candidates compete,. Same forward pass. Same cost. The architecture cannot tell the difference.

Consider a model generating a Python function. After for i in range(len(my_list, the next token is near-deterministic, any model trained on a week of GitHub data predicts the closing parenthesis ) with near-certainty. Then comes the second closing bracket ), the colon :, the newline, and the indent. Each one burning full frontier compute to predict syntax a rule-based system could have handled.

Then the logic lands. The one token that actually matters. Same price as every boilerplate token that preceded it.

That compounds in multi-agent systems. Every agent pays full model cost for every token, including the ones that never needed a frontier model in the first place.

This is the flat latency problem. You cannot fix it by scaling hardware.

Speculative Decoding: Run Ahead, Verify Fast

Not every token deserves a frontier model. So we should stop sending them all to one and use a hybrid strategy. Speculative decoding works like this.

A small, cheap Draft Model runs ahead of the main model. It generates K tokens speculatively — fast, low cost, imperfect. The large Verifier Model then checks all K drafted tokens in a single parallel forward pass. Not K passes. One.

If the draft tokens match what the verifier would have generated, you have produced K tokens for the cost of roughly one verification pass. You have paid once and received K. That is the efficiency gain.

If they diverge at token N, you accept tokens 1 through N-1, grab the verifier’s corrected token for position N, discard the rest of the draft, and let the drafter resume. You have lost nothing in output quality. The final sequence is always what the large model would have produced. The drafter never contaminates the output. It only accelerates the path to it.

If the draft model is accurate on 7 out of every 10 tokens, your effective generation speed increases dramatically without compromising model quality, model size, or hardware. The verifier is still the brain. The drafter is just the leg work.

One important caveat. The drafter has to be fast enough and accurate enough to pay for its own overhead. A drafter that’s wrong on 8 out of 10 tokens means you are running a second model for almost no gain. The draft acceptance rate is the number to watch. If it falls below roughly 60%, you are adding complexity without recovering the latency.

Why Nobody Shipped This in Production — Until Now

Speculative decoding has been supported in HuggingFace for years — a single assistant_model argument in generate() is all it takes technically. The problem was never the API. It was finding a drafter that actually works with your model.

The draft model and verifier have to share the same tokenizer and output distribution. In practice that means same-family models, or a distilled or quantized version of the verifier itself. Pair the wrong two models and your acceptance rate collapses — you are running two models and gaining nothing. Most combinations don’t work. So most teams didn’t bother.

The operational question was always: which smaller model is actually compatible with mine, and who maintains that pairing as the verifier gets updated? There was no clean answer.

Gemma 4 changed everything!

Everything above assumes you have a drafter worth trusting. Gemma 4 ships with one.

Its built-in MTP (Multi-Token Prediction) drafter is not a separate model bolted on — it is prediction heads trained directly into the architecture. That distinction matters: the drafter shares the target model’s KV cache and activations, so it never recalculates context the larger model already has. This is not just operationally cleaner than a DIY pairing. It is architecturally faster.

No compatibility hunting. No distribution drift. No second model to maintain. The result is up to 3x faster inference with zero degradation in output quality.

The pairing problem that quietly killed speculative decoding in most production stacks has been absorbed by the model itself. If you are running Gemma 4 and not using its drafter, you are leaving the one optimization the model was specifically designed to deliver on the table.

Gemma 4 26B on a NVIDIA RTX PRO 6000. Standard Inference (left) vs. MTP Drafter (right) in tokens per second. Same output quality, half the wait time.

The Dose Verdict

Flat latency is not a configuration problem. It is baked into the architecture of every transformer-based LLM ever deployed. Speculative decoding is the most meaningful structural answer we have today — and Gemma 4 is the first widely available model to ship it as a native capability, not a DIY assembly project. Its MTP drafter is the mechanism; speculative decoding is what it enables. No configuration, no compatibility hunting, no second model to maintain.

But this is one step. The deeper challenge, building models that allocate compute dynamically based on actual task difficulty, remains open. Speculative decoding gets us closer but the game is not over. It is just getting interesting.

📖 If you want to move from wherever you are today toward the Craft Engineer side of this chart, that is exactly why I wrote The Agentic AI Book — a production-first guide to building AI systems that actually work.

Grab early access: book.ryanrad.org

Until next dose — Dr. Ryan Rad