Seven Production Lessons for Building a Reliable Voice AI Agent
Latency, interruptions, tool failure paths, business logic outside the prompt, memory, observability and human handoff: what a voice agent needs at scale.
This article is published in English.
Sources: 7 Things I Wish I Knew Before Building a Production Voice Agent
Making a voice agent talk is surprisingly easy. Wire up telephony, speech-to-text, an LLM, text-to-speech and a handful of tools, and the result can hold a conversation. The real work begins when strangers start calling.
On a production agent taking hundreds of calls a day, trouble shows up that a demo never reveals. Callers interrupt and talk over the agent. They have accents nobody tested. APIs slow down. The LLM picks the wrong action. A tool call dies midway through a conversation. And a delay of a second or two that seemed harmless in development suddenly makes the agent feel broken. Seven lessons stand out for anyone about to build one.
1. Latency is the whole experience
This is probably the biggest lesson. In an ordinary app, a request taking a second or two is tolerable. In a spoken conversation it feels completely different. Picture a caller asking to book an appointment for tomorrow and then hearing nothing. One second, two, three. By then the caller is wondering if anyone heard.
A voice agent stacks several stages that each add delay:
Caller → Telephony → STT → LLM → TTS → Caller
Callers feel the total latency and never see any single component's share. Vapi, for one, describes voice-to-voice latency as the sum of transport, endpointing, transcription, model response and speech synthesis.
That changes how models should be picked. The most capable model isn't automatically the best for voice, and a slightly weaker one that answers sooner often yields a far better conversation. Voice agents should be tuned for time-to-first-word and not for raw model intelligence alone.
Streaming matters hugely too. The wrong flow waits for each stage to finish:
STT finishes → LLM finishes → TTS finishes → play audio
The better flow overlaps them:
STT streams → LLM streams → TTS streams → audio starts playing
The aim is something that feels like a conversation and not a chain of API calls.
2. Interruptions are normal
The assumed hard part is getting the LLM to write good replies. It isn't. The hard part is behaving naturally when the person doesn't follow the script. Real callers cut in with things like "Yeah, but wait", "No, not that one" or "Actually, can we do Friday instead?"
A production agent has to notice that the caller started talking and stop its own audio at the right moment. That involves:
- Voice Activity Detection, or VAD, to spot speech
- endpointing, deciding when a caller has finished
- detecting a barge-in while the agent talks
- managing who has the floor
- buffering audio sensibly
- cutting speech synthesis off and resuming it cleanly
Voice platforms now ship dedicated controls for this, since poor timing can make a clever agent feel awful. The balance is delicate. Interruption detection that's too sensitive lets background noise cut the agent off, while one that's too conservative has the agent talking over the caller.
Natural conversation depends on knowing when to speak and when to stay quiet, and not only on what's said.
3. Every tool needs a failure path
Rarely does a voice agent exist just to talk. Sooner or later it has to act: check an appointment, find an order, update a CRM, book a meeting, send a confirmation. That's where function calling enters, and the architecture grows to something like this:
At that point the agent stops being only an AI problem and becomes a distributed systems problem. Say the caller asks for a 4 PM slot tomorrow. The LLM calls checkAvailability(), the API takes two seconds, then the booking API fails, and the agent still has to tell the caller what happened. Throwing an exception and ending the request isn't an option. The agent must recover in conversation. Rather than a raw "ERROR 500", it needs wording like "That slot can't be checked at the moment. Want to try another time?"
Every tool the agent can call needs a failure path. Designing only the happy path from request to success isn't enough. The design should cover success, timeout, invalid input, API failure, partial failure, retry and fallback.
4. Keep business rules out of the prompt
Another trap is the ever-growing system prompt. It begins with a rule like "if the user says X, do Y", then picks up "but if X comes after Y, do Z", then "unless the question is about..." Soon a 5,000-word prompt carries half the application's business logic. It works until it doesn't.
A prompt should cover the role or identity, communication style and principles, boundaries and guardrails, conversation behavior, when to use tools and what information to collect. Deterministic business rules belong in the application. Whether an appointment can be booked, for example, shouldn't rest entirely on the LLM. The backend should decide:
LLM
↓
checkAvailability()
↓
Backend validates availability
↓
Booking API
↓
Result returned to agent
The LLM chooses what action to attempt, and the application decides whether that action is valid. Reasoning is for the LLM, and anything that must be correct is for deterministic code.
5. Context isn't memory
Having context doesn't mean a voice agent has memory. Several kinds of information are in play. What happened in this call, such as "the user wants an appointment on Friday"? What's known about this caller from earlier interactions, such as "the user usually prefers afternoons"? What should the agent always have on hand, such as "our cancellation policy is 24 hours"?
These shouldn't automatically be dumped into one giant prompt. Two questions need answers: what does the agent need right now, and what should persist after the call ends? This grows more important as conversations lengthen, since more context doesn't mean better context. Giving the model less but more relevant information sometimes produces better results. Memory doesn't mean storing everything. It means deciding what's worth retrieving at the right moment.
6. Observability, or you're guessing
Manual testing hides this problem. One call sounds good, the next one does too. Then production brings hundreds of calls and someone reports that "the agent was really slow". Where? Telephony, STT, endpointing, the LLM, tool execution, TTS or the network? Without real observability it's guesswork.
Early on, a team may evaluate an agent by simply listening to calls. That helps, but it's costly and doesn't scale, especially when running evaluations across many calls adds up. Eventually a separate observability dashboard becomes necessary so patterns surface without inspecting every conversation by hand.
At minimum, a single call should be understandable at a glance:
Call ID
├── duration
├── transcript
├── latency per turn
├── STT latency
├── LLM latency
├── TTS latency
├── tool calls
├── tool failures
├── interruptions
├── errors
└── final outcome
Debugging changes completely. "The agent felt slow" turns into "average LLM time-to-first-token rose from 350ms to 1.4s", and that's something fixable. A bad call that can't be inspected can't be reliably improved.
7. Know when to hand off
This may be the most important lesson. The goal isn't an AI that handles everything. It's a successful overall system, and sometimes the best action is a transfer to a human. Imagine a frustrated customer who's already repeated themselves three times. Pushing the AI to keep trying isn't intelligence. It's usually a terrible experience.
Production systems need clear boundaries, for instance:
Simple request
↓
AI
↓
Can AI solve it?
↙ ↘
Yes No
↓ ↓
Solve Fallback
↓
Human / callback
A fallback isn't the system failing. It's part of the design. A production agent shouldn't try to win every conversation, and it should recognize when to pass one along.
Reliability is the feature
Before building one, the central challenge seems to be making the AI sound intelligent. Afterward, a better question emerges: how can the whole system be made reliable enough that a person can trust it? That's a different engineering problem. It covers latency, turn-taking, telephony, APIs, failures, memory, observability and fallbacks, and also the LLM, which is just one component.
Getting a voice agent to talk is easy. Getting one through hundreds or thousands of real conversations is another challenge, and the gap isn't mainly a better prompt or a smarter model. It's the engineering wrapped around the model.
Before trying to make an agent "more intelligent", these seven areas deserve the focus:
- Latency
- Interruption handling
- Reliable tool execution
- Deterministic business logic
- Context and memory
- Observability
- Graceful fallbacks
Once an agent hits production, reliability becomes the feature.