Behind the Voice Agent

On paper a voice agent is three boxes in a row: speech-to-text, an LLM, text-to-speech. Build one and you find out how much sits between those boxes. The agent cuts people off mid-sentence, or leaves a two-second gap that makes it sound broken, or answers a question the caller was still in the middle of asking. Getting a machine to hold a conversation that feels natural turns out to be a problem of turn detection, utterance boundaries, and milliseconds.

This talk goes through where those milliseconds actually go and how you get them back. We’ll walk the latency budget of a live agent end to end, look at the architectural decisions that buy speed, and be honest about what each one costs you in quality. We’ll also compare the cascaded speech-to-text > LLM > text-to-speech pipeline against speech-to-speech models: what each approach is good at, where each falls down, and how to decide which one your use case needs.

Along the way we’ll look at real agents built for very different jobs, and why designs that solve the same problem end up looking nothing alike. Examples come from ElevenLabs agents, but the constraints are the same whatever stack you’re on.