Home / Articles

Voice AI Agent on Live Phone Calls: Why Asyncio Beat Threads, Plus Three Bugs

Lessons from a real-time STT, LLM and TTS phone agent: moving barge-in from threads to asyncio, plus leaked tool calls, dangling messages and stale transfers.

Publisher: REACTAPP.TOP7 min read1538 words

This article is published in English.

Sources: Building a Production Voice AI Agent: Threading, Async, and the Bugs Nobody Warns You About

A real-time voice agent that answers live phone calls chains speech-to-text, an LLM and text-to-speech, and plugs into a telephony system. It also needs barge-in, so a caller can interrupt the bot mid-sentence, plus transfer to a human agent and a clean disconnect. This is a look at the design choices and bugs that taught the most while running such a system under concurrent calls, beginning with a mistake that was nearly left unfixed.

The pipeline in brief

Each call runs three components in a chain:

Caller audio → STT (Deepgram, streaming) → LLM (Groq, LLaMA 70B) → TTS (Cartesia) → Caller
  • Speech-to-text streams audio over a WebSocket and returns transcripts, with voice-activity-based end-of-turn detection.
  • The LLM streams tokens back, and they are held until a sentence boundary (a period, question mark, exclamation mark or newline) before moving to speech synthesis, so the bot can begin talking before the whole reply exists.
  • Speech synthesis sends audio back to the caller over the same telephony bridge, paced to real time by sending 20 ms frames on a wall-clock schedule.

Around that sit barge-in detection, which halts the bot as soon as the caller talks over it, memory that summarizes older turns so that a 45-minute call stays inside the context window, and tool calling for actions such as handing off to a person or hanging up. No single component is the interesting part. What matters is what happens when many calls run at once on modest hardware.

Threads felt natural but weren't

The first design gave each call a background thread for barge-in. It listened to the caller's audio continuously and raised a flag when speech began, and the other stages consulted that flag to learn when to go quiet and start listening. Each call needs an always-on listener, so one thread per call seemed intuitive.

It fell apart quickly. On a laptop with 16 GB of RAM and an 8-core i5, four simultaneous calls were enough to make everything stutter, with dropped audio frames, sluggish interrupt detection and choppy playback.

The first suspicion was pool sizing: with a cap of 30 threads, call number 50 would queue behind others. That's a genuine concern at scale, yet it couldn't explain trouble at four calls.

The real cause was more basic. The workload is bound by waiting, not computing. Even a lightweight per-call listener spends nearly all its life idle, waiting for audio frames from the bridge and for enough signal to decide between speech and noise. The actual work, checking a flag or computing an RMS value for silence detection, takes microseconds between long pauses. OS threads are a poor fit for that. Each one carries a stack, kernel scheduling and context-switch costs, and Python adds the GIL for anything that isn't purely waiting on I/O. Every call pays thread overhead to sit around until the next frame.

Async/await turns this around. One event loop can hold thousands of tasks that are waiting on the network, because a suspended coroutine costs almost nothing: no dedicated stack, no kernel switch, just a place in a queue until its socket has data.

So the barge-in listener, along with the speech-to-text, LLM and synthesis layers, moved onto asyncio. The listener rewrite is a good illustration. Previously a background thread woke every 20 ms to poll a queue, checking for audio and judging whether it was speech, which was busy-waiting even during silence. After, the audio-forwarding loop calls asyncio.wait_for(queue.get(), timeout=...), which fully suspends the coroutine, using no CPU, until a frame arrives or the timeout fires. The barge-in flag becomes ordinary shared state read at await points, replacing a value written from another OS thread at unpredictable moments.

The concurrency ceiling moved as a result. The question is no longer how many threads the OS can schedule but how many sockets can stay open, what the provider's rate limit is and how much network bandwidth exists, a far friendlier scaling curve for this kind of load. Worth admitting: the original worry about pool exhaustion at 50 calls wasn't wrong, but it was a secondary, scale-related concern. The primary reason async wins is architectural, since threads are the wrong abstraction for work dominated by I/O waits, whatever the pool size.

Three bugs worth knowing

None are exotic. All surface only when real concurrent traffic hits a real LLM provider.

1. Tool calls leaking as plain text

Some Groq-hosted LLaMA checkpoints occasionally leave the structured tool_calls field empty. Instead they emit a function-call-looking string straight into the text stream, which text-to-speech would then read aloud.

Writing a regex for every shape seen so far loses the race, because slight changes in whitespace, key order or quote style get through. The better approach treats any raw brace or suspicious tag in the stream as provisionally unsafe. It's held back from speech immediately and released only once it proves to be one of two things: a complete, parseable tool call (converted into a real dispatch and removed from spoken output) or not a tool call after all (returned to the stream as normal text). A safety valve helps too. If the held buffer grows past a size limit without resolving, treat it as a false positive, perhaps a customer name that contained a brace-like character, and flush it as text instead of stalling the reply forever. The lesson: when output format is ambiguous, don't enumerate failure shapes. Buffer and verify.

2. Dangling tool calls after an interrupt

If a caller interrupts while the LLM is partway through emitting a tool call, history can end up with an AIMessage holding tool_calls that never received a result. Groq's API rejects this on the next turn since the message order no longer makes structural sense.

Two layers handle it. First, an explicit rollback on interrupt removes the dangling AI message, its orphaned tool results and the user message that triggered it. A sanitizer then runs before every LLM call as a last line of defense, dropping any AI and tool message pairs that don't match. The rollback should catch the problem, and the sanitizer ensures a bug elsewhere can't corrupt the next turn.

3. A stale transfer on a call that had ended

Transfer-to-human and disconnect are separate tool calls. Because of the leak handling in bug 1, the model can legitimately trigger both in one turn, one as a structured call and the other recovered from leaked text. If a transfer request was recorded earlier for another reason and the call then ends by disconnect, a downstream system asking whether the call was escalated might see the stale entry and send an already-finished call to a human queue.

The remedy is a precedence rule applied everywhere a call can end: disconnect always wins, and every path that ends a call first clears any pending transfer record. It's a small rule that only became obvious after watching the race happen, because two async tool dispatches can finish in either order depending on how the model formats its reply.

Unglamorous details that matter

A few production touches changed how the bot feels to talk to:

  • Sentence-boundary buffering. Text goes to synthesis in whole sentences, not token by token, which makes speech sound natural instead of a stutter of fragments.
  • A startup guard on barge-in. Interrupt detection ignores roughly the first one and a half to two seconds of the bot speaking, because some synthesis providers buffer the entire clip before streaming. During that silent window the microphone is already open, and a quick reply to the previous turn can be mistaken for an interruption of the current one.
  • Voice-activity tuning of the end-of-turn threshold and timeout. These numbers separate a bot that cuts people off mid-thought from one that feels slow, and no universal value exists, so they're tuned against real call recordings.

What it took

None of this needed exotic infrastructure, with no Kubernetes, custom C extensions or bespoke networking. It took asyncio, careful state management and a readiness to trace what happens when two async tasks race. Provider-agnostic abstractions for LLM, speech-to-text and synthesis, and a clean split between the conversation engine and the telephony transport, made it possible to swap providers, add tools and fix these bugs without rewriting everything. For anyone building something similar, the first question is whether the workload is bound by I/O or by CPU before reaching for threads. It could save a baffling afternoon of watching four calls choke a capable laptop.