Why AI works worse in your language

Home/Blog/Why AI works worse in your language

July 25, 2026 · Artifipedia · Language, Multilingual, Equity

Why AI works worse in your language

The gap between English and everything else is not about linguistic difficulty. It is about training data share, tokenizer fitting, and where instruction-tuning stopped, and only one of those three is expensive to fix.

A model that was trained overwhelmingly on English can answer in Japanese, translate into Swahili and write code comments in Portuguese without anyone teaching it those languages as separate tasks. That is surprising, and it is real. It is also, for most of the world's languages, not good enough, and the reasons it is not good enough are more mundane and more fixable than they first appear.

The performance gap between English and other languages has almost nothing to do with how hard those languages are. It comes from three things: how much of that language existed in the training data, how efficiently the tokenizer handles its script, and whether instruction-tuning covered it at all. Two of those are accidents of how the systems were built rather than facts about language, which means they are correctable, and one of them is quietly making non-English use three to four times more expensive.

This is an explanation of where multilingual capability comes from, why it degrades, what the degradation actually costs, and which of the fixes are real.

The gap, measured

It helps to be concrete about size, because "works less well" understates it considerably.

On enterprise retrieval tasks, accuracy drops of up to twenty-nine percent have been measured between English and non-English queries against the same underlying documents. That is not a subtle degradation. That is a system that answers correctly nine times in ten in English and roughly six or seven times in ten elsewhere, with the same corpus and the same model.

At the far end of the resource distribution it stops being a gap and becomes a floor. Evaluations on indigenous languages of the Americas, absent from the training data of the multilingual models tested, have produced accuracy around thirty-eight percent on a three-way classification task. Chance is thirty-three. The model is not performing poorly in those languages; it is performing at approximately the level of guessing.

Between those two poles sits most of the world. The pattern is consistent: performance tracks resource availability, and resource availability tracks how much of a language exists in scrapeable text on the public internet, which correlates with wealth and with colonial history rather than with speaker population. Bengali has more speakers than German and a fraction of the digital footprint.

Why speaker count does not predict anything

It is worth dwelling on the mismatch, because it drives most of the inequality.

Bengali has roughly a quarter of a billion speakers and a digital footprint smaller than that of several European languages with a tenth the population. Hausa, Yoruba and Igbo together cover a substantial share of West Africa and appear in scraped corpora at a rate that would suggest they are minority languages spoken by hobbyists. Meanwhile Icelandic, with a few hundred thousand speakers, is comparatively well served because Iceland is wealthy, connected, and has invested deliberately in language technology.

What the corpus measures is not how many people speak a language. It measures how much text in that language has been published on the open web, which is a function of internet penetration, literacy in the written standard, the existence of a publishing industry, whether the language has a settled orthography, and whether its speakers write in it online or code-switch into a colonial language when they type.

That last one is underappreciated. In many multilingual societies people speak one language and write another, or write their own language in a romanised form that no standard corpus recognises. The digital record systematically understates what people actually speak, and the models inherit that understatement.

What actually predicts performance

Three factors, none of which is linguistic complexity.

Training data share. The dominant one. A language's representation in the pretraining corpus is roughly its representation in scraped web text, which is wildly unequal. The standard taxonomy of this inequality sorts the world's languages into tiers, and the top tier contains a handful of languages while the bottom contains thousands with almost nothing.

Tokenizer efficiency. Covered in detail below, because it is the least understood and the most immediately expensive.

Instruction-tuning coverage. Pretraining is broad; the instruction-tuning that turns a text predictor into something that follows requests is usually much narrower. This produces a specific and confusing failure: a model that clearly understands your language, and answers a question in it fluently, while ignoring the actual instruction you gave. Comprehension transferred. Instruction-following did not.

What is not on this list is worth stating. Morphological complexity, script direction, agglutination, tonality: none of these reliably predict how well a model performs. Finnish and Turkish are morphologically demanding and models handle them reasonably, because there is enough Finnish and Turkish text. The difficulty is a data problem wearing a linguistics costume.

The tokenizer tax

This is the part that surprises people who have been working with these systems for years, and it is the clearest example of a fixable accident being mistaken for a property of language.

A model does not read characters. It reads tokens, produced by splitting text using a vocabulary of subword pieces learned from a training corpus. The algorithm counts which adjacent pairs of characters appear together most often and merges them into single units, repeating until the vocabulary reaches its target size. Common English words end up as single tokens. Everything else gets fragmented.

Because the merges were learned from text that was overwhelmingly English, the resulting vocabulary encodes English efficiently and everything else poorly. The consequence is a direct multiplier on cost and a direct reduction in usable context. The same sentence, carrying the same meaning, can consume three or four times more tokens in one language than another, and in the worst cases considerably more.

Three groups are hit hardest. Non-Latin scripts, because their characters were rare in the fitting corpus and often decompose into multiple bytes before any merging happens. Agglutinative languages, which build long words from many morphemes, where a single semantic unit shatters into many pieces. And romanised text, which is neither the original script nor standard English, and falls into a gap between the two distributions the tokenizer actually learned.

The costs compound in three directions at once. You pay more per request. Your effective context window shrinks by the same factor, so a model advertising a large context offers substantially less of it to some users than others. And performance itself degrades, because the model sees the text at a coarser effective resolution, with meaning spread across more positions than the equivalent English.

Systematic evaluation across large numbers of languages has found that tokenizer quality is a strong predictor of downstream task performance, independent of everything else. That is a striking result. A preprocessing decision, made once before training and frozen thereafter, accounts for a meaningful share of how well a model works in a given language.

The mitigation that has actually been adopted is larger vocabularies. Recent model families have moved to vocabularies several times bigger than the earlier norm, which leaves room for more non-English merges and reduces fragmentation. This helps and does not solve it, since vocabulary space is finite and the allocation still follows the training distribution.

Where the capability comes from

Given all that, the surprising thing is that cross-lingual performance is as good as it is. A model never explicitly taught to translate can translate. A model fine-tuned on a task in English often improves at that task in other languages it was never fine-tuned on.

The leading account is that these models develop a partially shared representation space. Semantically equivalent sentences from different languages end up in similar regions of the model's internal state, with language-specific processing concentrated near the input and output and more abstract, less language-bound processing in the middle. Evidence comes from probing studies that find parallel sentences clustering together, and from the transfer effect itself, which is hard to explain otherwise.

This has a practical implication people underuse: prompting in English and requesting output in the target language frequently beats prompting in the target language. The instruction is understood more reliably in the language the instruction-tuning covered, and generation into the target language is a separate and better-supported capability. It feels wrong and it often works.

There is a serious caveat, which the frontier section returns to. Recent work has found that stronger representational alignment between languages does not consistently produce better performance. If sharing a representation space were the whole mechanism, more sharing should mean more transfer, and it does not straightforwardly. Something is going on that the simple picture does not capture.

Where it breaks

The failures cluster in specific places rather than being uniform degradation.

Cultural and local knowledge. A model can be fluent in a language and ignorant of the world that language describes. Local institutions, regional history, culturally specific terms, the entities that matter in that context: these appear in proportion to their presence in the training data, which for most languages means barely. Translation-based evaluation misses this entirely, because a translated benchmark tests whether the model handles the language, not whether it knows the place.

Reasoning tasks specifically. The gap is larger on multi-step reasoning than on factual recall, which suggests the model's internal reasoning process is more anglophone than its knowledge is. Interestingly, this pattern does not hold uniformly: on some factual and real-world query distributions the gap narrows or behaves differently, which complicates any simple story.

Fluency without accuracy. This is the failure mode most likely to reach production undetected. Non-English output is often perfectly fluent and confidently wrong. Fluency is cheap for these models; correctness in a low-resource language is not. A reviewer who does not speak the language cannot tell the difference, and automated metrics that reward fluency will not catch it either.

Safety and moderation. Guardrails trained on English content transfer imperfectly. This runs in both directions, producing both under-blocking in languages the safety training did not cover and over-blocking of ordinary speech that pattern-matches to something flagged in English.

The evaluation problem underneath all of this

A quieter difficulty runs beneath every claim in this piece: we are not very good at measuring the thing.

Most multilingual benchmarks are translated from English. That makes them comparable across languages, which is why they are built that way, and it means they test a narrow thing: whether the model handles the language, holding the content constant. They cannot test whether the model knows anything about the world that language is spoken in, because the content came from somewhere else.

The result is a systematic blind spot. A model can score respectably on a translated benchmark in a language while being ignorant of that language's context, and the

[truncated for AI cost control]