Flagship models are not always better

Frontier Models Suck Sometimes (A Benchmark)

Written on Jul 8, 2026 • 18 minute read • Share on 𝕏

I’ve been building a small realtime note taking app. It’s like one button: you tap it, you talk on your iPhone or from your wrist on the Apple Watch, and it hands your rambling thought back to you as text plus like a summary, some action items you buried, and some questions you left dangling. On-device speech-to-text, then “AI analysis,” then explore.

The word carrying the entire app on its back in that sentence is AI.

The “AI” that was counting words

I decided to start small: the first version’s “AI” was just a heuristic, algorithm-based solution (not generative). It counted words. The “topics” it proudly surfaced were, literally, just the most frequent tokens in your transcript. So it would inform you, with total confidence, that today’s themes were “know” and “it’s”. I looked at it and was like bruh this isn’t AI, it’s Array.prototype.sort.

At this point I was like okay, maybe I don’t need regex for this but like a real language model. But it had to be on-device and private. It also needed to process loooooooooong conversations, like 2-hour podcast episodes and things. How can we process that when language models often come with smaller context windows?

Hierarchical map-reduce

A language model can only look at so much text at once: a budget called its context window. Apple’s on-device model tops out around 4,000 tokens (where a token is roughly 1 english word). Even a small Qwen model that could fit on an iPhone (I downloaded it) isn’t much roomier: a 27-minute ramble is many times that. So you run straight into a genuinely strange question: how do you summarize something that doesn’t even fit inside the reader’s head?

Picture summarizing a fat book when you can only hold a few pages in your mind at a time. You wouldn’t try to swallow it whole, you’d read one chunk, jot the key points on an index card, and move on. Chunk, card. Chunk, card. Do that for the whole book and you’ve quietly turned one impossible task into a stack of very possible ones. In other words,

Map: slice the transcript into bite-sized pieces that do fit, and run the model on each piece on its own. Each pass pulls out just that slice’s key points, action items, and questions. (It’s called “map” because you apply the same little function across every piece. This is the same concept as array.map().)

Reduce: now gather all those index cards into a pile and write the final answer from the cards, not from the original book. Merge the duplicates, drop the overlap, fold it all into one clean set plus a title and a summary. (“Reduce” just means folding many things down into one. This is the same concept as array.reduce().)

Hierarchical: when there are so many chapters that even the pile of index cards won’t fit on the table, group the cards, summarize each group onto a new card, and do it again: summaries of summaries until the whole thing finally fits. Like folding a long letter in half, and in half again, until it’s pocket-sized. A short note never needs that extra fold, a genuinely multi-hour recording does, and this is what keeps the whole thing from tipping over.

Apple Intelligence said no

The tasteful, on-brand move for a privacy-first notes app is Apple’s Foundation Models. Real LLMs on the device without talking to the cloud that just work offline. Wow. I wired it in behind the analyzer, kept my chunked map-reduce pipeline, and fed it a real recording: a genuinely private, 27-minute conversation with my wife about random stuff. Half an hour of half-formed thoughts about moving countries, a birthday, the price of a Tesla in Berlin, the people at last week’s meetings. The exact kind of sprawling human talk this app exists to make sense of.

Apple’s model looked at it and returned:

Detected content likely to be unsafe.

At this point, even though there was nothing unsafe in it, I turned the safety dial all the way down to Apple’s most permissive setting (permissiveContentTransformations) and it still refused.

Apple Intelligence is such garbage. I just want to transcribe a voice note on device from a regular conversation and it refuses. Absolute trash.

— Tejas Kumar (@TejasKumar_) View tweet

A guardrail that fires on a private note about a birthday isn’t a guardrail but instead some bullshit that ruins people’s experience. I needed something better.

So I downloaded a brain

Pivot: open weights. Qwen3-4B-Instruct-2507, Apache-2.0, quantized to a 4-bit GGUF. This weighs (pun intended) in at ~2.3 GB and runs fully on-device through llama.rn. Same hierarchical map-reduce pipeline, but I just swapped the model using react-native-ai. No cloud, no key, no guardrail. It analyzes whatever you record, because it’s yours and it never leaves. It works.

Getting 2.3 GB of model to actually run on an iPhone was two good battles.

Battle one: memory

On the CPU it was so slow the app looked hung to the point where you start checking the debugger to see if you’ve deadlocked yourself. Move it to the GPU (n_gpu_layers: 99, full Metal offload) and it flies, but mapping ~2.5 GB into resident memory gets you instantly jetsammed on an 8 GB iPhone. The fix is a single entitlement, com.apple.developer.kernel.increased-memory-limit. This btw is Apple quietly conceding that your phone can, in fact, hold other language models in addition to its own. 🤫

Battle two: the model thought itself to death

Qwen3 is a “thinking” model. Its chat template opens a block and reasons before answering. Delightful, except my token budget is finite and it spent the entire thing thinking and never got around to emitting the JSON. Literally like some people, “all thinking no doing”. Every note came back empty. The one-line fix was to turn off the thinking:

chat_template_kwargs: { enable_thinking: false; }

Then it started to work! However while the little 4B on my phone produced genuinely structured insight, private and free and offline, the data was really not usable: it understood the conversation, but produced pretty trashy summaries with even worse action items and questions. This was more noise than signal from my conversation. Eventually, I decided I need a bit more power: it’s time to consider maybe possibly using cloud models.

The Benchmark

In 2026, is this the best we can do? Or am I shipping a cute compromise and calling it a feature? Locally, the ceiling unfortunately was the Qwen model I tried (for an iPhone 16 Pro Max): it had high scores in benchmarks and looked SOTA (State of the Art) for the iPhone level. This is of course did not include cloud models. In an effort to find the best of the best without the hardware constraint, I did a benchmark. Here’s how I did it.

Methodology

The data. One input, held constant: the transcript of that same 27-minute voice note. Every model saw the identical text. I’m benchmarking the analysis, not the transcription, so the speech-to-text step is upstream and the same for everyone. Whatever imperfections the transcript had (and it had some, it’s a real recording), everyone shared them.

The candidates. 11 models:

My on-device Qwen3-4B, plus 10 cloud models via OpenRouter,

Anthropic’s Claude Sonnet 5

Opus 4.8,

OpenAI’s GPT-5.5 Pro,

Google’s Gemini 3.1 Pro,

xAI’s Grok 4.3,

DeepSeek’s V4 Pro,

Qwen3-Max-Thinking,

Kimi K2.6,

z-ai’s GLM-4.7, and

GLM-5.2.

The task. Every model got the exact same prompt, asking for structured JSON so the outputs were directly comparable:

You analyze a spoken voice-note transcript into structured insights. Return ONLY a JSON object with EXACTLY these keys: {"title": string (, every single time, and the model only ever sees the placeholders. The map from back to “Peter” never leaves the device, so only your phone can turn the answer back into real names. This is the layer that saves you even if every other layer fails. A totally compromised vendor sitting on raw plaintext just sees owes some slides, and has no clue who that is. Now, is it perfect? Nope. On-device recognition will miss a name here and there, or misread a weird one. So I keep it conservative, keep the map on the phone, and treat it as one layer of defense, not a magic eraser. (If the built-in recognizer isn’t good enough, tiny redactor models like OpenAI’s open-weight Privacy Filter are a later upgrade).

Cover the boring channels too. These are mostly cheap and mostly soon. Batching and padding requests on a slightly random schedule, and stripping identifying headers, are all app-level stuff that rides along with the gateway work anyway (OHTTP leaves timing and size correlation out of scope, so mild batching is the sensible patch). Encrypted Client Hello is a CDN toggle, not app code. A boring, standard TLS fingerprint I get for free just by using the phone’s normal networking stack, so I blend into the crowd.

1. Encrypt everything the app stores

Everything the app stores (audio, transcript, the insights, even the titles and timestamps) should get encrypted on the device before it ever touches disk, so my own backend only ever holds ciphertext. The interesting question is where the key comes from.

To begin, I make a random 256-bit key per note, seal each payload with XChaCha20-Poly1305 (there are solid, maintained crypto libraries for React Native, or I lean on SQLCipher down at the database layer), and wrap the master key with a Secure Enclave key that never leaves the chip and unlocks with Face ID. That alone gets me encrypted-at-rest, hardware-backed, on this device. It also sits on a data model that Proton and Standard Notes have already battle-tested: one random key per note, each wrapped by the master key. I encrypt the metadata too (titles, tags, timestamps), because Signal’s whole sealed sender lesson is that locking the letter but printing the address on the outside is only half a job.

Eventually we can go multi-device. The WebAuthn PRF extension lets a passkey cough up a deterministic 32 bytes of secret from a Face ID gesture, which I run through HKDF into a key. The neat part is that a passkey synced through iCloud Keychain carries that secret to every device you own, so the same key just shows up on your iPad and MacBook and your notes decrypt themselves. No key server, no me in the middle. I love it. It’s also the newest, least-trodden path on this whole list, so it waits until the basic version is rock solid. Since a passkey that fails to sync would take your notes down with it, I’d like to ship it with a printable recovery key so losing every device isn’t the same as losing your notes. The end state is basically Apple’s Advanced Data Protection: keys live only on your trusted devices, recovery is a code you hold, and the provider (me) simply can’t read your stuff (ref: ADP).

1. Don’t just trust the vendor, verify it can’t read you

My winning model, Kimi K2.6, is open-weight, and in 2026 open-weight models can run inside a GPU confidential-computing enclave that you can actually verify. Phala lists Kimi K2.6 on Intel TDX plus NVIDIA H100, behind an OpenAI-compatible endpoint. The model runs in hardware-encrypted memory that’s sealed even from the machine’s own operator, and the enclave hands back a signed attestation quote proving exactly which code and which weights are about to touch your transcript (ref: NEAR AI, Tinfoil). TLS terminates inside the box, not at some load balancer that could peek. The old complaint about all this was speed, and that’s basically gone now: overhead is down around 95 to 99% of native.

This is probably gonna be a longer effort, but initially for the MVP I plan to start with OpenRouter’s one-click Zero Data Retention, routed only to endpoints that don’t store or train on your data, prompt logging off. It’s a config flag and really more of a promise than a proof, but it’s a decent start imo. The provider still sees plaintext while it’s working, and you’re trusting them to keep the contrac

[truncated for AI cost control]