Towards local plug-and-play AI

May 17, 2026

Last week I wrote about the hardware side of running AI locally, why memory bandwidth matters more than raw compute, which machines are worth building, and where the market is heading. If you missed it, start there as this post builds directly on top of it.

In the quest of becoming AI independent, your hardware sets the ceiling, but what decides how close you actually get to it is software.

Two machines with identical GPUs, identical VRAM, identical bandwidth, one running naive inference, one running an optimised stack can produce a 3-5x difference in tokens per second. This can mean the difference from running at 5tok/s to the 20-30tok/s that you need to get something usable. Even more, some techniques, and software-model-hardware optimised implementations may allow you to fit large models like DeepSeeekV4-Flash on a MacBook with at least 96GB of RAM. Same model, same hardware, different choices in the software layer (I feel we have a lot to learn from the video encoding and compression industry in this respect. We have to squeeze the most of every $ of hardware resources).

Last week I presented my new goal to create an inference box that generates tokens fast enough within a certain price point. So far I haven’t found one that fits my needs fully. What I am hoping is that the outcome of this work finds me a hardware configuration optimised for local inference and that is plug-and-play and not absurdly expensive, and/or a tool that detects your current hardware and suggests the best model and configuration for it.

This post continues that search focusing on the latest techniques to improve my inference stack.

MoE vs dense models

Before we get to the software tricks, there’s an architectural decision that sits underneath all of them, because it changes what the software layer has to deal with.

Most of the interesting models you’re likely to run locally with a decent throughput are Mixture-of-Experts architectures. Qwen3.6-35B-A3B, Qwen3-235B-A22B-250, DeepSeek-V4. The naming convention tells you the structure: 35B total parameters, but only 3B active per token. The model is divided into expert sub-networks, and a router that decides which ones fire for each token.

The key advantage of these types of models (and why is the one with biggest changes to fit your hardware) is that If only 3B of 30B parameters do any work per token, you get something close to 3B-scale inference speed while the model carries 30B-scale knowledge. With the right serving trick, like llama.cpp’s -ngl 99 -ncmoe 99 flags which keep the attention and shared weights on the GPU and offload cold expert FFN layers to system RAM, a Qwen3.6-35B-A3B can hit 33.5 tok/s on an RTX 3070 Ti with just 8GB of VRAM, provided you have 64GB+ of fast system RAM for the offloaded experts. The floor for running a 35B-knowledge model just dropped further than most people realise.

The main downside is consistency. And if you have used one of these MoE long enough you have probably experienced what I am about to describe.

Think of a MoE model as a hospital. Each patient gets routed to the right specialist. But which specialist fires depends on the token. The model can feel sharp in one domain and noticeably weaker in another depending on which expert activates. Dense models don’t have this problem because every parameter processes every token, every time. Slower, more expensive, but completely consistent. This lack of consistency can be experienced through tool call loops, to performance degradation and catastrophic forgetting.

For reasoning tasks, for long-context coherence, for anything where you need the model to stay sharp across a 50,000-token context, dense models tend to be better. This is why I would always recommend dense models for any agentic task that requires several assistant turns and accurate context keeping.

There’s a serving problem too: when too many tokens in a batch route to the same expert simultaneously, that expert’s buffer overflows and tokens get dropped silently. The model will not warn you of this happening, and it just gets worse. Dense inference has none of that complexity.

So how do you choose between dense and MoE models? Here’s the practical decision tree that I currently use myself:

8GB VRAM GPU + 64GB system RAM? MoE with expert offload is your only real option for a capable model. Qwen3.6-35B-A3B at Q4 or Gemma4-26B-A4B with llama.cpp offload fits this profile. Throughput will be CPU-bandwidth-bound, not GPU-bound, so a fast DDR5 system matters more than GPU generation here.

16–24GB VRAM (RTX 3090, RTX 4090, RTX 4080)? you have a genuine choice. Dense Qwen3.6-27B at Q4_K_M fits in ~16GB with no offloading, no serving complexity, consistent quality. MoE Qwen3-35B-A3B also fits at this tier with partial offload. If your workload is agentic, i.e. long contexts, tool calls, multi-step coherence, etc. The dense model’s consistency advantage is worth the slightly lower raw throughput (which can be painful as described in my last post).

128GB+ unified memory (Mac M3 Ultra, Strix Halo / Ryzen AI Max+)? This is where MoE becomes unambiguously better. You can hold the entire Qwen3-235B-A22B (235B total, 22B active per token) in unified memory and run it at full context without any offloading. At this tier you get frontier-class capability locally. The 22B active parameters per token still give you strong throughput on unified memory’s high-bandwidth pool. And sharp readers may be wondering, but then why are you running dense models like Qwen3.6-27B on your Strix Halo yourself? We are back to the consistency problem. I am looking to run long-running agentic tasks. The performance quickly collapses as the context grows.

Multi-GPU / 192GB+ VRAM? You can essentially run decent models for whatever you need. MoE at full precision, no quantisation required. The routing overhead becomes negligible at this level of parallelism.

The short version: if you have limited VRAM and a lot of system RAM, MoE with expert offload. If you have enough VRAM to fit a dense model cleanly, prefer dense for anything requiring long-context coherence. If you have a large unified-memory machine, MoE at the top tier, e.g. Qwen3-235B-A22B. becomes the most capable option you can run locally.

Fortunately, Alibaba built both ends of this spectrum with Qwen3.6, and so far I’ve been using them and I am decently happy. The MoE option is Qwen3.6-35B-A3B. The dense option is Qwen3.6-27B. Both run the same hybrid attention core, the choice between them is almost entirely a hardware and workload question, not a model quality one.

How did I get to these numbers? As I was doing the research for this section, I came across LocalMaxxing.com. This site is just pure gold. It provides benchmarks of different models over different hardware architectures, with clear information about the inference engine used and their configuration. This gave me homework for the next few months, and is a great resource for this new quest of mine.

The attention zoo

The MoE/dense question is only one dimension of the model architecture. The other is which attention mechanism the model uses internally. Every variant is a different answer to the same constraint: standard attention produces an N×N matrix where N is sequence length. At 10,000 tokens that’s 100 million entries, and the memory cost scales quadratically. Every interesting development in attention over the last two years is essentially a different attempt to escape that curve without paying too much in quality. Each have its own trade-offs and may fit better different underlying architectures.

Sebastian Raschka’s visual guide is the best resource I can recommend to navigate the attention landscape. The progression is worth following because each step reveals a different tradeoff.

Standard Multi Head Attention is what everyone (or maybe it’s me) thinks of when thinking about the attention mechanism in a transformer, where every token attends to every other token, full stop. Quadratic memory, quadratic compute. Nobody building for local hardware chooses it today; it’s just the baseline everything else is measured against.

GQA (Grouped-Query Attention) was the first fix that actually shipped at scale. Instead of each query head maintaining its own independent key-value projection, multiple query heads share a single KV pair. Roughly 50% KV cache savings, almost no quality loss. Llama 3, Qwen3, Gemma 3 all use it. The tradeoff is minimal, this is why it became the default so quickly.

MLA (Multi-Head Latent Attention), from DeepSeek, goes deeper. Rather than reducing how many KV pairs you keep, it compresses what’s stored in each one, saving a latent representation and reconstructing the full KV state on demand. More complex to serve than GQA, but at scale the quality-per-byte advantage is real. DeepSeek V3 and Kimi K2 use it; it’s the right choice when you have the memory to absorb the reconstruction overhead and want frontier-class output quality. Don’t ask me why, but I personally love models that implement MHA.

SWA (Sliding Window Attention) takes a different angle entirely. Rather than compressing the cache, it simply restricts how far back each token looks. Gemma 3 uses a 5:1 ratio of local to global layers, with a 1,024-token window and GQA on top. Memory grows linearly rather than quadratically, and for most practical workloads (like code completion, document Q&A, chat) the quality impact is minimal. The tradeoff is that you’re genuinely giving up global context on those local layers. Fine for most tasks; matters for very long-range reasoning.

Gated DeltaNet, used in Qwen3.6-27B, pushes further: rather than attending over a stored sequence at all, it maintains a fast-weight memory that gets continuously updated with each new token. Memory footprint stays flat regardless of sequence length. Going from 4k to 65k context costs ~800MB of VRAM instead of several GB, which is the difference between a 16GB card hitting a wall and staying competitive with machines three times its size. The tradeoff is architectural complexity and the fact that very-long-range dependencies that would be trivial for full attention require the model to have learned to compress them into the running memory state.

Mamba-2 hybrids (Nvidia’s Nemotron Nano) are the logical extreme, where most attention is replaced by recurrent state machines with constant memory regardless of sequence length. The right choice for edge and embedded hardware where even GQA is too much. Not the first option when you have a proper GPU available; the quality ceiling is lower, but the memory floor is the lowest in the landscape.

These differences show up on the machine. With SWA at long contexts the KV cache barely moves, the serving engine has headroom it wouldn’t have with MHA. Switch to GQA at the same context length and VRAM climbs noticeably. Run a DeltaNet hybrid and the memory profile nearly flatlines. When you’re fitting multiple agents on a fixed memory budget, which attention variant the model uses matters as much as how many parameters it has.

Even one more dimension on top of this that I’ve decided to leave out of this post and deserve its own post, is which quantisation mechanism to use and the different flavours available (which is a beast in itself). See my post about TurboQuant for a primer on this.

Speculative decoding

A few weeks ago Google announced that Gemma 4 was shipping with dedicated MTP drafters: small companion models that could push inference speed up to 3x without touching quality.

The problem speculative decoding solves is a fundamental one of transformers. As we’ve described a few times in this newsletter, LLMs generate text autoregressively, i.e. one token at a time, each token depending on everything before it. The large model has to do a full forward pass for every single token. You can’t parallelise the generation itself. So no matter how fast

[truncated for AI cost control]