Why Voice Agents Sound Robotic: Cascades, Turn Detection and the TEN Framework
Why voice agents sound robotic: cascaded STT, LLM and TTS pipelines cause dead air and bad interruptions. Four kinds of voice agent and how TEN handles turns.
This article is published in English.
Sources: Why Your Voice Agent Sounds Like a Robot on Hold and How TEN Fixes It
Most voice agents that feel bad do not have a model problem. The language model is usually clever enough. What goes wrong is timing: the agent talks over a person who was only thinking, stays silent for three seconds after a question, or refuses to stop when someone cuts in. This walk-through explains why the classic way of building voice agents produces those symptoms, which kinds of voice agent exist, and how the open-source TEN framework from Agora approaches the timing problem.
What counts as a conversational agent
The term gets used so loosely that it has lost most of its meaning, so a plain definition helps. A conversational agent is software that holds a live, two-way dialogue and grasps the meaning of what is said, as opposed to matching keywords. The word agent adds the ability to act on that understanding: look up a record, book a meeting, update a ticket, or answer using context from earlier in the session.
A phone menu that asks you to press 1 for billing is not one. A support bot that hears someone say they have not received the refund for last Tuesday's order and replies with the status of that specific order is close. At its best, such an agent deals with interruptions gracefully, remembers the session, listens and speaks at the same time as people do, and answers quickly enough that silence is never mistaken for a malfunction. Speed and naturalness are exactly where most systems fall apart.
The cascade and why it feels awkward
Before frameworks like TEN, a voice agent meant a cascading pipeline of three separate systems in sequence: speech-to-text, then a language model, then text-to-speech. The person speaks, the first model turns audio into a transcript, the second writes a text reply, and the third turns that reply into audio.
On paper it is tidy. In practice each stage waits for the one before it to finish completely. Transcription sends nothing until the speaker stops. The language model does not start until the whole transcript has arrived. Speech synthesis does not begin until the full reply exists. A naive build easily adds two to four seconds of dead air before anything is heard, and that silence reads as a bug.
Guessing when someone is done
A cascade has no smart way to tell when a person has finished. It relies on voice activity detection, which really just listens for quiet. If the gap exceeds a threshold, perhaps half a second, the system assumes the turn is over and begins processing. The consequences are familiar:
- A speaker who pauses mid-sentence to think gets cut off by an early reply.
- A speaker who wants to interrupt the agent finds that it keeps talking.
- The exchange is structurally turn-based, like texting out loud.
- With separate vendors for each stage, debugging means tracing across three systems.
A newer option and what it costs
Speech-to-speech models take audio in and return audio in a single pass. GPT-4o Realtime and Gemini Live work that way. Latency is lower because no text sits in the middle. The price is modularity: the reasoning part cannot be swapped independently of the voice, a compliance filter cannot easily be inserted between input and output, and these unified models tend to be weaker at complex reasoning and tool calls than dedicated language models.
Four kinds of voice agent
Before picking a framework, it helps to decide which kind of agent is being built, since each maps to different user expectations.
The listener. Strictly half-duplex, with one speaker at a time. The agent waits to confirm silence and then answers. Classic phone menus, voice form-fillers and command assistants such as setting a timer live here. Easy to build, robotic to use, and fine when the task is structured. It breaks the moment the conversation gets nuanced.
The interrupter. Full-duplex agents listen and speak simultaneously and cope with interruptions without losing context. A mid-sentence correction actually makes the agent stop. This needs intelligent turn detection and not just silence detection, and it is the baseline for anything that should feel natural.
The specialist. Multimodal agents handle video, images, documents and data streams alongside the speech. Examples include a tutor that watches a screen while a student talks about code, a medical assistant that studies an X-ray as symptoms are described, or a meeting bot that summarizes live. These need a framework able to coordinate several streams at once without a bottleneck.
The collaborator. Multi-party agents join group settings: meetings, classrooms, multiplayer games, language lessons with several learners. They need speaker diarization to know who is talking, context across every speaker and the ability to address particular people. On a naive pipeline this is close to impossible, and it is far more approachable on a framework with real-time audio routing built in.
What TEN is
TEN stands for Transformative Extensions Network. It is an open-source framework from Agora for real-time, multimodal conversational agents. Put simply, it is a runtime that sits between the AI models and the user's ears. It orchestrates the streaming, the routing of audio among models and the state of the conversation, so that none of that has to be assembled from glue code.
It was open-sourced in 2024 and has since been used for AI companions, translators, live support bots and real-time meeting summarizers. A hosted version, Agora's Conversational AI Engine, adds the company's global Software-Defined Real-Time Network underneath, so the infrastructure is already managed.
A useful mental model: a cascading pipeline means managing three vendor relationships and hoping data flows correctly between them. With TEN, the parts are extensions, modular and swappable inside a single graph that the framework runs. Extensions are building blocks. Cascade stages are separate buildings joined by footbridges.
What sets it apart
A graph that runs in parallel
In a cascade each stage waits. TEN runs its pipeline as a directed graph in which every extension, whether transcription, language model, synthesis or voice detection, is a node. The nodes work concurrently and pass partial output forward as it appears. This is how the framework claims to cut latency by 60 to 70 percent against naive builds: recognition, generation and synthesis are all streaming at once.
A model that knows when to speak
TEN ships its own turn detection model, built on Qwen2.5-7B and trained for conversational intent. Rather than noticing silence, it classifies each utterance into one of three states:
- Finished. The thought is complete and a reply is expected, as in a request for help with an order.
- Wait. The person has told the agent not to speak, with something like hold on.
- Unfinished. There was a pause, but more is coming, so the agent stays quiet.
Understanding why a pause occurred, not merely that it did, is what separates this from silence-based detection. English and Chinese are supported natively.
A cheap filter in front of transcription
A lightweight voice activity model also acts as a pre-filter. Before audio reaches speech-to-text, it removes background noise, music, breathing and silence. Less data goes to the transcription service, which cuts API cost directly. Used together, the two give a two-layer system: one decides whether speech is happening, the other decides whether the turn is over.
Swappable parts
Every component is an extension. Moving from one transcription vendor to another means changing one extension. Comparing GPT-4o with DeepSeek in the language layer means swapping extensions. Adding retrieval so the agent can draw on company documents means dropping in a memory extension. Custom extensions can be written in Python, Go or C++, and the runtime works on Windows, Mac, Linux and mobile platforms.
A visual designer
TMAN Designer is a visual tool for assembling agent graphs without code. Extensions are dragged, connected and configured through an interface, and the resulting graph can be inspected before deployment. That widens the circle beyond backend engineers, since product teams can prototype configurations and pass them along.
Edge and cloud together
Extensions can run on the device near the user, in the cloud, or in a mix. Latency-sensitive work stays at the edge and heavy reasoning goes to the cloud. Balancing cost and speed becomes a configuration change and not a rewrite.
A global network underneath
Through the hosted engine, the agent gets access to Agora's real-time network, which routes audio dynamically, copes with packet loss and keeps latency low across regions. The same infrastructure carries large volumes of ordinary audio and video calls, so the agent behaves well under real network conditions and not just in a quiet demo.
When it fits and when it does not
TEN is a good match when the goal is natural, real-time conversation and not question-and-answer turns, when LLM, transcription and synthesis vendors should be mixed freely, when the experience is multimodal, when global scale is needed without building WebRTC infrastructure, and when interruption handling must not feel broken. Typical targets are AI companions, live tutors, nuanced customer support and collaborative meeting tools.
Alternatives are better in three cases. If the absolute lowest latency matters more than modularity, a speech-to-speech model is the better pick. If the use case is a simple command and response with no interruptions, a basic cascade is enough. And if every input must be transcribed and stored at every stage for compliance, a cascaded system with a text boundary is more auditable.
Turn detection is the product
The main lesson after studying all this is that most voice quality problems are not language model problems. They appear at the edges: cutting a person off too early, failing to notice they are still talking, or continuing to speak after being interrupted. The graph and the hybrid deployment matter, but the most important contribution may be the dedicated turn detection model, built for the one problem that decides whether talking to a machine feels natural. Everything else can be tuned. If the agent does not know when to stop talking, nothing else counts.
The framework is open source under the MIT license, runs from a Docker container in a few minutes, and includes a working sample agent that is worth running before writing any custom code. The best voice agent is the one whose architecture a user forgets about because the conversation simply works.