AI STS Stack for underserved languages paper

Jul 06, 2026

The demo works until you switch the language

Every real-time voice AI demo you’ve seen runs in English. The latency is magic, the turn-taking feels human, and the vendor’s pricing page has a calculator that makes your CFO happy. Then you change one parameter — the language your users actually speak — and the entire stack evaluation you did last month is worthless.

I found this out building an AI interview platform for a financial holding in Azerbaijan. Azerbaijani has around 24 million speakers, a Latin script, and clean Turkic morphology. It is not an obscure language. It is, however, a low-resource language in the only sense that matters here: the amount of transcribed speech data in the training corpora of every major STT and speech-to-speech model.

That single fact reshapes the whole architecture decision:

Speech-to-speech models (OpenAI Realtime, Gemini Live) give you the best latency and the most natural conversation — but you’re locked into whatever language coverage the model has, and you can’t swap components when it doesn’t.

Cascaded pipelines (LiveKit, Pipecat, Vapi with custom providers) let you plug in any STT/TTS you can find — but now you own the latency budget, the turn-detection logic, and the integration of whatever Azerbaijani speech models exist.

In high-resource languages this is a preference. In low-resource languages it’s a fork: one path may simply not work at all, and the other may work badly in ways you only discover in production, with a real candidate on a real phone line.

This post is the evaluation I wish had existed before I started: what actually breaks, in which stack, and the decision framework that generalizes to any language the demos weren’t made for.

The speech-to-speech path — great, until it isn’t

Start with the option everyone wants to work: a single model that takes audio in and produces audio out. OpenAI Realtime and Gemini Live. No pipeline to assemble, sub-second responses, natural interruption handling for free.

The appeal is real. In English, both are the closest thing to a solved problem this space has. So the first question isn’t “which is better” — it’s “does either one actually speak Azerbaijani, or does it just claim to?”

There are distinct failure levels, and vendor docs only tell you about the first:

Not listed. The language isn’t on the support page. Easy — you’re done.

Listed but degraded. The language is “supported,” and comprehension is fine on clean, slow speech — then falls apart on phone-quality audio, regional accents, code-switching into Russian mid-sentence (which every real Azerbaijani speaker does), or domain vocabulary. This is the level that kills you in production, because your demo will pass.

Asymmetric. The model understands the language but speaks it with a foreign accent, wrong stress patterns, or drifting phonology. For an interview platform, a robot that sounds foreign is a credibility problem with candidates.

What I found:

OpenAI Realtime: Failed at level 3. Comprehension was workable, but the spoken Azerbaijani came out with a distorted accent — wrong phonology, wrong stress. For casual use, tolerable. For a system conducting job interviews at a financial institution, disqualifying: candidates hear the accent in the first sentence, and it sets the tone for everything after. There’s no voice tuning or pronunciation control that fixes this — the failure is in the model, and the model isn’t yours.

Gemini Live: The surprise. Comprehension and output quality were genuinely good — it passed all three levels. What killed it was a fourth failure mode my own taxonomy missed: latency. Response delays were long enough to break conversational rhythm. In a chat interface that’s a spinner; in voice it’s the candidate wondering whether the call dropped, talking over the model, and the turn-taking collapsing from there. Real-time voice has a hard perceptual budget — cross it and quality elsewhere stops mattering.

So the scoreboard for speech-to-speech and Azerbaijani, mid-2026: one model that speaks the language badly, and one that speaks it well but too slowly for conversation. Neither failure is fixable from my side of the API. That’s the structural bet you make with speech-to-speech — when it works, it’s the best option available; when it misses your language on any dimension, you’re waiting on someone else’s roadmap.

Which forces you down the other path: assembling the pipeline yourself.

The cascaded path — you own every millisecond

If speech-to-speech fails your language, the fallback is the classic pipeline: voice activity detection → speech-to-text → LLM → text-to-speech, stitched together by an orchestration layer. The pitch is control: every component is swappable, so if one link doesn’t speak Azerbaijani, you replace that link instead of abandoning the stack.

The catch is that the pitch assumes replacement parts exist. So before comparing orchestrators, do the component hunt — because if there’s no usable Azerbaijani STT or TTS, the orchestrator comparison is decoration.

For Azerbaijani, mid-2026, the hunt is more encouraging than it was even a year ago:

TTS: Azure Neural TTS has had generally available Azerbaijani voices for years — the workhorse option, enterprise-friendly, unexciting but reliable. ElevenLabs also supports Azerbaijani, including regional accent adaptation, at a premium price point and a noticeably higher naturalness ceiling.

STT: Historically the weak link — Whisper technically includes Azerbaijani but its low-resource performance is a lottery, and code-switching into Russian confuses everything. The situation changed materially when ElevenLabs’ Scribe brought Azerbaijani into a 99-language ASR model, and the Scribe v2 release in early 2026 added a real-time variant aimed specifically at agentic use cases — real-time being the non-negotiable property for a voice agent.

So the components exist. What you’re buying with a cascaded stack is the right to use them — and what you’re paying is ownership of the latency budget. Every handoff adds delay: endpoint detection, STT finalization, LLM time-to-first-token, TTS time-to-first-byte, network hops between all of them. Speech-to-speech models hide this sum inside one model; in a pipeline, you engineer it — streaming every stage and overlapping everything that can overlap — to get back under the conversational threshold Gemini couldn’t hit.

The orchestrator choice is then mostly a question of how much of that engineering you want to own:

Vapi: managed, fastest to demo, least control over exactly the low-level knobs (custom STT/TTS providers, turn-taking tuning) that a low-resource language forces you to turn.

LiveKit Agents: infrastructure-grade WebRTC plus an agent framework; the most control and the most surface area to operate.

Pipecat: open-source pipeline framework; maximum flexibility, community-maintained connectors, you run everything.

I ended up on LiveKit. Two configurations survived testing: Deepgram STT + Cartesia TTS around Gemini 2.5, and an all-ElevenLabs pipeline around Gemini 2.5 Flash — both conversationally solid. The decider was cost: the setup leaning on on-prem components runs roughly $0.02 per 10 minutes of conversation, while the ElevenLabs stack lands around $0.90 per 10 minutes — a ~45× spread that stops being academic the moment you interview at scale.

The general rule that fell out of the exercise: the fewer languages your target has in the training data, the further down the stack you’ll be forced to go. High-resource languages get to buy the abstraction. Low-resource languages have to build below it.

The checklist — evaluating any stack for a language the demos ignore

Compressing the above into the protocol I’d run on day one, for any language:

Test speech-to-speech first, but test it adversarially. Not the happy path — phone-quality 8 kHz audio, fast speech, regional accents, code-switching into the local second language, your domain vocabulary. Score each model on four failure levels: not listed, listed-but-degraded comprehension, asymmetric output (understands but speaks badly), and latency. A pass on the vendor’s language-support page is level zero of four.

If any level fails, check whether it’s fixable from your side of the API. For speech-to-speech, the answer is almost always no — and an unfixable failure, at any level, means the path is closed regardless of how good the rest is.

Run the component hunt before choosing an orchestrator. Find at least one production-viable STT and TTS for your language, and confirm the STT has a streaming/real-time mode — batch-only transcription quality is irrelevant for conversation. Re-run this hunt every few months; the low-resource landscape is moving fast, and a conclusion from last year is probably stale.

Budget latency end-to-end, not per component. Set a voice-to-voice target under about a second, allocate it across endpoint detection, STT, LLM first token, and TTS first byte, and reject any component that can’t stream. A pipeline of individually fine components can still be conversationally dead.

Weight switching cost, not just current quality. The best answer today is a model update away from being wrong. The cascaded stack’s real product isn’t quality — it’s the option to swap any link without rebuilding. In a low-resource language, that option is worth the latency engineering it costs you.

The meta-lesson: language support isn’t a boolean on a pricing page. It’s a distribution over failure modes, and for most of the world’s languages, you only find out where you land by testing with real audio from real speakers. The demos were never made for you. Build accordingly.

Model behavior and vendor language support described here reflect my testing as of mid-2026; both change fast — re-verify before you rely on any of it