Building Voice Agents: Cascaded Pipelines, Speech-To-Speech, and the War on Latency

In a text chat, a two-second pause is fine. Nobody notices. In a phone call, two seconds of silence feels like the line dropped. People start saying "hello?" and talking over the bot, and the whole conversation collapses.

That is the thing nobody tells you about voice agents: the LLM part is the easy bit. The hard part is time. Humans take turns in conversation with gaps of a few hundred milliseconds. Your agent has to hear, think and speak inside a budget that is not much bigger than that.

A voice agent is a real-time system that happens to contain an LLM, not an LLM app that happens to talk.

Two Architectures

1. Cascaded: STT → LLM → TTS

caller audio → VAD + turn detection → speech-to-text (streaming) → LLM (+ tools, RAG) → text-to-speech (streaming) → audio back to caller — The cascaded (a.k.a. pipeline or chained) architecture. Three specialised models, glued with streaming.

Analogy: a UN interpreter booth with three people. One writes down what is said, one decides the reply, one reads it aloud. Each is excellent at their job; the risk is the hand-offs.

2. Speech-To-Speech (Native Audio Models)

caller audio → single multimodal model (audio in → audio out) → audio back to caller — One model hears and speaks directly. Realtime audio APIs from the big labs work this way.

Analogy: one bilingual person who just talks to you. No hand-offs, and they can hear your tone — but you cannot easily see their notes.

Cascaded vs Speech-To-Speech

Why Cascaded Is Still the Production Default

Control: you see the transcript and the text reply. You can log it, filter it, run guardrails and evaluate it.
Best-of-breed parts: swap STT for a better one for Hindi, TTS for a better voice, LLM for a cheaper one — independently.
Tools and RAG: the LLM step is a normal text LLM, so function calling, retrieval and structured output all just work.
Cost and debuggability: when something goes wrong, you can tell which stage failed.

Where Speech-To-Speech Wins

Latency: no hand-offs, often the lowest time to first audio.
Paralinguistics: it can hear hesitation, emotion, laughter, and respond in kind.
Naturalness: prosody and interruptions feel more human.

Where It Hurts

Less control and observability — the "reasoning" happens in audio space.
Harder to run guardrails and deterministic business logic mid-utterance.
Fewer choices of voice, language and model; vendor lock-in.

My rule: if the agent does business (bookings, support, payments), start cascaded. If the agent is the experience (companions, language practice), try speech-to-speech.

The Latency Budget

What users feel is voice-to-voice latency: time from when they stop speaking to when they hear the first syllable back. In a cascade, every stage below adds to it. Exact numbers depend entirely on your providers, models and regions — so instrument each stage and measure your own.

end of speech detected (endpointing) → final transcript → LLM time-to-first-token → first sentence ready → TTS time-to-first-byte → network → speaker — Every box is a place latency hides. Optimise the biggest one first.

Endpointing (deciding the user has finished): often the single biggest, hidden chunk — hundreds of ms of silence waiting.
STT finalisation: small if streaming, large if you send the whole clip at the end.
LLM time-to-first-token: depends on model size, prompt length, and whether the prompt prefix is cached.
TTS time-to-first-byte: fast streaming TTS starts speaking before the sentence is fully synthesised.
Network: every hop between regions adds up. Co-locate STT, LLM and TTS if you can.

Optimisation Basics

1. Stream Everything

The naive cascade waits for each stage to finish. The fast cascade overlaps them like an assembly line: STT emits partial transcripts while the user speaks, the LLM streams tokens, and TTS starts speaking the first sentence while the LLM is still writing the second.

partial transcripts while user talks → final transcript → LLM streams tokens → sentence splitter → TTS speaks sentence 1 while LLM writes sentence 2 — Streaming turns a sum of latencies into (mostly) the latency of the first chunk.

python

async def respond(transcript):
    buffer = ""
    async for token in llm.stream(messages + [user(transcript)]):
        buffer += token
        # flush on sentence boundaries so TTS can start early
        if buffer.rstrip().endswith((".", "?", "!")) and len(buffer) > 20:
            await tts.speak(buffer)
            buffer = ""
    if buffer.strip():
        await tts.speak(buffer)

Chunk on sentence (or clause) boundaries, not single tokens — TTS needs a bit of context to get the intonation right.

2. Turn Detection: Know When They Are Done

VAD (voice activity detection) tells you if someone is speaking. It does not tell you if they have finished their thought.
Pure silence timeouts are a trade-off: too short and you interrupt people mid-sentence ("my number is 98… 45…"), too long and the agent feels slow.
Semantic turn detection: a small model looks at the transcript and audio and predicts "is this turn complete?". Lets you use a short timeout when the sentence is clearly done, and wait longer when it is not.

3. Barge-In: Let Them Interrupt

Humans interrupt. When the user starts talking while the agent is speaking, you must stop TTS playback immediately, cancel the in-flight LLM generation, and — importantly — trim the conversation history to what was actually spoken, not what was generated. Otherwise the agent "remembers" saying things the user never heard.

agent speaking → VAD: user started talking → stop audio + cancel LLM → truncate history to spoken text → listen — Barge-in handling. Echo cancellation matters here, or the agent interrupts itself.

4. Make the LLM Step Fast

Use the smallest model that passes your evals. Voice rewards speed over brilliance.
Keep the system prompt stable so prompt/prefix caching kicks in.
Keep replies short: prompt for one or two sentences. Nobody wants a voice agent reading a paragraph.
Write for the ear: no markdown, no bullet points, spell out numbers and dates the way they should be spoken.

5. Hide Unavoidable Latency

Tool calls (database, booking API) can take seconds. Say a short filler first ("Let me check that for you…") while the call runs.
Speculative work: start retrieval or tool calls on partial transcripts, and throw the result away if the final transcript changes.
Pre-warm connections to STT, LLM and TTS before the call starts.

6. Transport

For browser and app clients, WebRTC handles jitter, packet loss and echo cancellation better than raw WebSockets. For phone calls, you are going through telephony (SIP/PSTN) and 8 kHz audio, which also affects STT accuracy — test with real phone audio, not studio recordings.

Putting It Together

WebRTC / telephony in → VAD + semantic turn detection → streaming STT → LLM (small, cached prompt, short replies, tools) → sentence chunker → streaming TTS → audio out (loop: user interrupts → cancel, truncate, listen) — A production cascaded voice agent. Barge-in can cut in at any point.

What to Measure

Voice-to-voice latency: p50 and p95, per stage.
Interruptions: how often the agent cuts the user off, and how often the user has to repeat themselves.
Task success: did the booking/support flow complete?
Transcript-level evals on the LLM step, exactly like any text agent — which is where cascaded pays off.

The best voice agents are not the ones with the smartest model. They are the ones where someone obsessively shaved 100 ms off every stage and made interruptions feel natural. Build the pipeline, instrument every hop, and then go hunting.