How to Build an AI Voice Agent in 2026: Architecture and Stack

An AI voice agent answers phone calls, works out what the caller wants from natural speech and either replies or takes an action, with no menu of numbered options in between. Three pieces cooperate to make that happen: a model that transcribes the caller's audio, an LLM that decides what to do about it and a model that speaks the answer back.

The phone call never went away. Companies steered support toward chat and email, and customers kept dialing anyway. A voice agent fills that gap. It's not a chatbot with a speaker bolted on. It listens, reads intent from how people actually talk and behaves much as a trained person on the line would.

Getting one past the demo stage means picking an architecture and a stack, then pushing latency under a second before a real caller ever touches it. The sections below go through all of that.

The build in eight decisions

Each choice narrows the options for the next one.

1. Define the call workflow. Decide which call types the agent owns, what it may do and exactly when a person takes over.
2. Pick an architecture. Cascaded, half-cascade or native speech-to-speech. The choice fixes your latency floor and cost ceiling.
3. Choose telephony. Twilio, AWS Connect or WebRTC, based on channel and region.
4. Add speech-to-text. Deepgram Nova-3 or AssemblyAI in streaming mode, with custom vocabulary for domain terms loaded before launch.
5. Connect the LLM and tools. Set up function calling and escalation rules, and link the CRM or booking system.
6. Add text-to-speech. Choose a voice, enable streaming and target first audio below 200ms.
7. Test latency and interruptions. Run scripted and adversarial scenarios and aim for under a second end to end.
8. Monitor after launch. Watch completion rate, containment, CSAT and handle time starting on day one.

Why companies are building them

The call volume that once justified a phone team no longer fits inside one. Repetitive questions, after-hours calls, no-show follow-ups and outbound reminders all arrive through the same channel, and hiring people to cover everything stopped scaling.

Interactive voice response menus lose callers, who hang up the moment they fall outside the options. A voice agent behaves differently, since it can hold a real conversation, ask follow-up questions, route complicated cases and carry out repetitive call workflows at scale. Adoption figures point the same way. Surveys put voice assistant usage among mobile users at 97%, 89% of customers say they prefer brands offering voice AI support and companies deploying it report operating costs 20 to 30% lower. Treat those as industry survey numbers, not guarantees.

The pressure tends to show up in five places where human teams can't keep up:

Repetitive inbound volume. In logistics, an estimated 50 to 70% of inbound calls are "where is my order" queries. The same question, thousands of times a day, handled by people who could be doing harder work.

After-hours calls. Patients try to book at midnight and loan applicants call on Sundays. Customers don't schedule their questions around opening hours.

No-shows with no recovery. Clinics lose roughly 20% of appointments to no-shows. Without automated follow-up, the slot stays empty and the revenue goes with it.

Manual outbound work. Renewal reminders, installment follow-ups and delivery confirmations all follow the same script. Paying people to read it aloud is expensive and unnecessary.

Spikes nobody can absorb. Ten agents handle ten calls. A voice agent handles thousands at once, with no queue and no hold music.

Three architectures

The architecture determines how fast the agent answers, how much control your team keeps, how easy debugging is and how predictable cost stays at scale. Three production patterns dominate, each trading latency, control and cost differently, and a wrong pick can be expensive to undo.

1. The cascaded pipeline

Three specialized models run in sequence. Speech-to-text turns audio into text, an LLM reads that text and writes a reply, and text-to-speech turns the reply back into audio. Plain text passes between the stages, and that simplicity is the whole point. It's the most widely used production architecture in 2026.

Realistic end-to-end latency sits at 500 to 800ms. That isn't the fastest, but it buys full modularity. You can swap Deepgram for AssemblyAI, ChatGPT for Claude or Cartesia for ElevenLabs without touching anything else. Every component produces logs, and every layer can be measured, tuned and replaced on its own. High-volume, telephony-heavy and compliance-sensitive deployments mostly run this way, not because it's the most sophisticated but because it's the most controllable.

A cascaded build breaks down into these parts, each independently replaceable:

• Telephony stream. Connect through Twilio or AWS Connect and pass the inbound audio on to speech recognition via SIP or WebRTC.
• Streaming STT. Deepgram Nova-3 in streaming mode starts transcribing before the caller has finished the sentence.
• LLM orchestration. Hand the transcript to GPT-5 or Claude Sonnet with a prompt defining scope, tone and escalation conditions.
• Function calling. Link the CRM, booking system or database. The model calls them mid-conversation and waits for the result.
• Streaming TTS. Send model output to Cartesia or ElevenLabs, so playback starts before the full response has been generated.
• Logging and replay. Capture transcripts and audio at every layer, so a failure can be traced to a specific component.
• Human handoff. Define which intents and keywords trigger a warm transfer to a live agent.

2. The half-cascade

Here audio goes straight into a multimodal model that hears it natively, and only the output side passes through a dedicated text-to-speech model. That one change brings latency down to 300 to 500ms, and it enables something the cascade can't do.

The model notices a frustrated caller, picks up rising intonation on a question or catches a language switch halfway through a sentence. No transcription step sits between the sound and the model's understanding. Output still goes through a specialized TTS model, so voice quality remains finely controllable. ElevenLabs Agents and Google's Gemini Live both follow this pattern, and it's the right choice when how someone says something matters as much as what they say.

3. Native speech-to-speech

One model handles everything: audio in, audio out, no text in between and no component boundaries. Latency falls to 200 to 300ms. Below 300ms a conversation stops feeling like software, which matters wherever hesitation erodes trust.

The trade-offs are real, though. On long calls, cost climbs because the model reprocesses every earlier token of the conversation at each turn, so a call that looks like $0.30 a minute can reach $1.50 in practice. You also give up component-level control. When something goes wrong, you can't isolate the layer responsible, because there are no layers.

One more limit applies. Phone calls over the public network carry 8kHz audio, which discards much of the richness native audio models are meant to exploit, so their edge over a cascade shrinks a lot.

Which to choose

Start with the cascade if the deployment involves telephony, compliance logging or a need for predictable cost. Modularity alone justifies it for most enterprise cases. Move to the half-cascade when input audio matters, such as callers switching languages or emotional tone detection, or anything where transcription throws away signal you need. Pick speech-to-speech only when responses under 300ms are non-negotiable and call volumes leave the per-minute economics workable. That covers a narrow set of cases, and for most deployments the cascade takes you further.

| Criteria                 | Cascaded Pipeline                         | Half-Cascade                            | Native Speech-to-Speech                    |
|--------------------------|-------------------------------------------|-----------------------------------------|--------------------------------------------|
| Latency                  | 500-800ms                                 | 300-500ms                               | 200-300ms                                  |
| Approx. Cost/min         | Low                                       | Medium                                  | High ($0.30 can reach $1.50)               |
| Modularity               | Full, swap any component independently    | Partial, TTS is swappable, input        | None, one model, no layers                 |
|                          |                                           | model is not                            |                                            |
| Emotion & Tone Detection | No, transcription strips it              | Yes, model hears audio natively         | Yes                                        |
| Compliance Logging       | Per-layer logs on every component         | Partial                                 | Not possible                               |
| PSTN Compatibility       | Strong                                    | Strong                                  | Weak, 8kHz audio narrows the               |
|                          |                                           |                                         | quality advantage                          |
| Best For                 | Telephony, compliance-sensitive,          | Multilingual callers, emotional         | Sub-300ms use cases with manageable        |
|                          | high-volume deployments                   | context, tone detection                 | call volume                                |
| Avoid When               | Latency is a hard requirement             | You need full component control         | Long calls, regulated industries,          |
|                          | under 500ms                               | or strict cost caps                     | unpredictable volume                       |

The five layers

However simple or elaborate the agent, the skeleton is identical: sound enters, an answer leaves, and the steps in between decide how fast, how accurate and how human the exchange feels.

| Layer         | Role                    | Common Tools                  | Enterprise Consideration          |
|---------------|-------------------------|-------------------------------|-----------------------------------|
| Telephony     | Call routing            | Twilio, AWS Connect, SIP      | Region coverage, number portability |
| STT           | Speech transcription    | Deepgram, AssemblyAI, Whisper | Accents, noise, vocabulary        |
| LLM           | Reasoning and response  | GPT, Claude, Gemini, Llama    | Latency, compliance, tool use     |
| TTS           | Voice output            | ElevenLabs, Cartesia, PlayHT  | First-byte latency, brand voice   |
| Orchestration | Turn-taking and routing | Retell AI, Vapi, LiveKit      | Barge-in, handoff, monitoring     |

The components, in order:

1. Telephony. The front door. A caller dials in, and this layer grabs the audio and sends it down the pipeline. Twilio, AWS Connect and WebRTC are the common choices.
2. Speech-to-text. Converts what the caller says into text. Noise, accents and specialist terms all affect quality, and any mistakes here echo through every later layer.
3. LLM. The brain. It reads the transcript, works out intent, picks an action and writes the reply. When it needs outside data, such as an order status or a balance, it calls a tool and waits.
4. Text-to-speech. Converts the reply back to audio, and both voice quality and responsiveness depend on it.
5. Integrations. Connections to the real systems: CRMs, booking platforms, databases and payment processors. Without them the agent can chat but can't do anything.

Each handoff adds latency and each component can fail independently. Knowing where each one sits is what turns a five-hour debugging session into a five-minute one.

Layer 1: speech-to-text

Everything downstream depends on this output, so errors here spread through the whole pipeline. The model takes an audio stream, splits it into chunks and predicts the likeliest word sequence from acoustic patterns and language probabilities. Modern STT is fast enough to transcribe word by word while the caller is still speaking.

What hurts accuracy. Four things degrade STT in production: background noise, strong accents, domain vocabulary and poor audio quality. The phone network adds to the problem, since its 8kHz ceiling removes the frequencies that help models tell similar-sounding words apart.

Tools.

• Deepgram Nova-3 is the usual pick for live telephony, handling noisy lines, accents and low bitrates without much fuss.
• AssemblyAI Universal-Streaming shines when the audio is clean.
• OpenAI's gpt-4o-transcribe is accurate, yet the added delay is felt during a live conversation.
• Whisper is a strong model that was designed for batch jobs, so it's a poor fit for streaming.

Streaming versus batch. Batch waits for the caller to finish before processing, while streaming transcribes as audio arrives. For voice agents, streaming is the only workable choice.

Vocabulary gaps. Out of the box, most STT models trip over brand names, medical terms and industry jargon. Custom word lists, pronunciation hints and domain fine-tuning fix most of that, so reserve time for it ahead of launch in healthcare, logistics or finance.

Layer 2: the LLM

The STT layer produces text, and the model decides what to do with it. It reads the transcript, identifies the request, picks an action and writes a reply. On a simple call that means answering a question. On a complex one it means calling an external tool, waiting and weaving the data into a natural answer, all in the time a human takes to pause between sentences.

Choosing a model. GPT-5 and Claude Sonnet are the most dependable picks for voice in 2026, since both manage context well, follow instructions closely and keep replies short enough to speak. Gemini 3 Flash deserves a look when latency is the priority. Llama and Mistral suit teams that need on-premise deployment or tighter cost control.

Voice prompts versus chat prompts. Voice has one constraint chat lacks, which is length. An agent that reads out a paragraph loses the caller. The system prompt should specify tone, reply length, escalation conditions and the exact wording to use when the agent has no answer.

Function calling. When a caller asks about an order, the agent queries the order system or CRM, waits for the result and then answers. That's what separates an agent that talks from one that acts. One catch: function calls block streaming, so design the conversation around the pause.

Memory. There are two kinds. Session memory covers a single call so nobody has to repeat details, and persistent memory spans calls. Teams generally ship session memory first and layer persistence on after the core pipeline settles.

Layer 3: text-to-speech

The caller's final impression comes from TTS. Even a correct, fast and well-judged reply loses the person if it sounds robotic.

What makes a voice sound natural. Prosody, pacing and first-byte speed. Prosody is the melody of speech, the pitch movement that tells a listener whether something is a question. Pacing is about pauses landing in the right spots. Models trained on huge amounts of real speech do both well now, and robotic output has become much rarer than it was in 2025. First-byte speed is the delay before the caller hears anything after generation starts.

Tools.

• Cartesia Sonic is the speed leader, with first audio arriving sooner than from most rivals.
• ElevenLabs is the most lifelike and offers cloning from a short audio sample.
• OpenAI's gpt-4o-mini-tts makes a sensible default if the rest of the stack is already OpenAI.
• PlayHT and Rime are worth a trial when a specific accent or regional tone is needed.

Voice cloning. Most platforms clone a voice from a brief recording, which gives enterprises a consistent brand voice on every call. A hospital can sound calm and clinical while a quick-service restaurant chain sounds fast and upbeat. Settle this before go-live.

Streaming TTS. The caller should hear the first word while the model is still writing the last sentence, which means audio goes out before the reply is complete. What counts is first-byte latency and not the total time to generate. Keep it below 200ms on live calls.

Layer 4: telephony and channels

This is where the outside world meets the pipeline. How audio travels from caller to STT model determines call quality, latency and how much infrastructure you control.

How calls connect. PSTN, the ordinary phone network, limits audio to 8kHz, which hurts every model after it. SIP sits on top for VoIP traffic and offers finer routing control. WebRTC travels over the internet with better audio, so it fits browser and app use. A common split is SIP for inbound phone calls and WebRTC for anything on the web.

Providers.

• Twilio is the safe default thanks to wide coverage, dependable SIP trunking and good docs, although it gets pricey at volume.
• AWS Connect fits teams that already live on AWS.
• Vonage is a good option for international rollouts, especially across Europe and the Middle East.

Orchestration platforms. Retell AI, Vapi and LiveKit occupy the middle, between the carrier and your STT, LLM and TTS choices. They manage live audio routing, turn-taking, interruption detection and handoffs. For phone deployments Retell AI is the most mature, Vapi gives the most room for custom pipelines and LiveKit does best on WebRTC.

Inbound and outbound. An inbound flow sits waiting for a call and responds as it arrives. An outbound flow dials out, verifies who answered and follows a scripted task like a payment reminder or an appointment confirmation. The pipeline is the same either way. Only the initiator and the opening differ.

Layer 5: integrations

Integrations are what let the agent act and not just talk. A model can read intent and draft a reply, but with no link to real systems it can't book a slot, look up an order or change a record. All it can do is describe what it would do.

What gets connected. Typically four categories: CRMs for looking up accounts and contacts, scheduling tools for bookings, order or logistics systems for status checks and payment processors for billing. A clinic's agent needs the scheduler, a shop's agent needs the order API. Systems vary, but the loop is constant: the model calls out mid-call, waits and uses the answer in its next sentence.

How the call works. Function calling drives it, the same mechanism described for the LLM layer. When the model decides it needs outside data it emits a structured request and stops. The result doesn't flow back by itself. Your own code has to catch it and append it to the conversation history before the model can use it. Without that step the agent hangs mid-call, holding nothing to say.

Where it breaks. Two spots cause most trouble. A slow API adds straight to the delay, and the caller can't tell whether the model is thinking or a CRM took 800ms to respond. A pause is a pause. The other spot is failure handling. When a lookup times out, the agent needs a prepared line to say, not silence and not a loop of repeated sentences.

Authentication. Credentials are normally API keys or OAuth tokens scoped to read-only or narrow write access. Checking order status only needs reads. Rescheduling appointments needs write access to a calendar, and that raises the stakes for error handling because a mistaken call now alters a live record.

Getting latency down

Three levers do most of the work: stream both STT and TTS instead of waiting for complete outputs, cache answers to frequent questions and keep every component in the same region. Together they trim perceived latency by around 40%, and the target to hold is under one second end to end.

Where the delay comes from

Each component adds its own slice. Streaming STT contributes 50 to 150ms, LLM inference adds 200 to 500ms depending on model and prompt length, and the first byte of TTS takes another 100 to 200ms, with network transit stacking on top at every hop. A pipeline that's merely average at each layer lands near 1.5 seconds before any optimization. Function calls make it worse, because inference pauses entirely until the tool returns, adding 300 to 800ms.

What to do about it

Streaming is the biggest lever. Run STT in streaming mode so transcription begins before the caller finishes speaking, and use streaming TTS so audio plays before the reply is fully written. Those two changes alone account for the 40% gain.

Caching helps too. If most callers ask the same three questions, pre-generate those answers and serve them instantly. And keep components together geographically. Splitting STT in US-East from an LLM in EU-West costs 80 to 120ms of plain network travel each turn.

For the lowest possible ceiling, native speech-to-speech brings end-to-end latency to 200 to 300ms. The cost trade-offs are significant, but the gain matters where conversational pace is critical.

Barge-in

Barge-in detection listens for caller audio while TTS is playing, cuts the reply off mid-sentence and reroutes right away. Retell AI and Vapi both support it natively. Tune the audio threshold so background noise doesn't trigger false interruptions.

Where to deploy

Deployment choice affects cost, compliance, latency and how much infrastructure you control, and it matters more than most teams realize until they're already in production.

Cloud is the default. AWS, Azure and GCP provide managed services for every pipeline component, with fast setup, automatic scaling and no hardware. The price is that data leaves your environment, which matters in regulated industries.

On-premise means audio, transcripts and customer records never leave your own servers. Healthcare, finance and any country with data residency rules often require exactly that. Open-source pieces like Whisper and self-hosted LLMs make it feasible without rebuilding every layer.

Hybrid deployment divides the pipeline by how sensitive each part is. Telephony and STT stay on-premise so raw audio never leaves, LLM inference goes to a managed cloud and TTS lives wherever delay is smallest. Enterprises often choose it because compliance reviewers can approve each layer's data handling separately.

Compliance

• For US healthcare, HIPAA means signed Business Associate Agreements with each vendor, encrypted audio at rest and access logs.
• GDPR covers every pipeline that touches EU residents, with clear consent notices for recorded calls.
• Data residency rules in the UAE, Saudi Arabia and Germany limit where conversation data can sit, so check each vendor against them before launch.

Scaling

Cloud deployments scale horizontally, while on-premise setups need capacity planning ahead of known peaks. Test at three to five times the expected peak volume before going live.

Monitoring

Four things deserve constant attention: call completion and transfer rates, component-level latency across STT, LLM and TTS, failed tool calls, and unhandled intents along with transcripts kept for replay.

If the transfer rate spikes, the conversation design is failing somewhere. If LLM response time keeps climbing, suspect a prompt that's grown too long or a tool call stuck waiting. LangSmith handles LLM observability, and Datadog or Grafana cover infrastructure.

Testing and evaluation

Most failures can be found before launch if the right things get tested.

1. Begin with scripted scenarios across your 20 to 30 core call flows.
2. Move on to adversarial testing: interrupt the agent, give it incomplete information and ask things outside its scope.
3. Watch for the failures that matter, which are confident wrong answers, handoffs that never trigger and tool calls that leave the line silent.
4. Use tools like Hamming or Vapi's evaluation suite to simulate hundreds of scenarios programmatically.

Four metrics matter most: task completion rate, containment rate, CSAT and average handle time. Track all of them from day one. Components that hold up at 10 concurrent calls often fail at 100, hence the three-to-five-times load test. After launch, review transcripts every week and use them to refine prompts, tighten function definitions and widen fallback handling. The agents that improve fastest have a structured review routine behind them.

Ten lessons from real deployments

Developers new to voice usually begin with one belief: pick a good model, write a solid prompt and ship. Six weeks later they're hunting dropped calls and latency spikes, puzzled that the agent states made-up facts with total confidence. These lessons are what teams learn the hard way.

1. Transport beats model choice. Swapping WebSockets for UDP can help more than changing the LLM. Most models do fine with a decent prompt, whereas latency and infrastructure don't respond to prompting at all.
2. One agent, one job. For several distinct tasks, run several agents that can talk to each other. A single agent stretched across everything cooperates poorly.
3. Function calling blocks streaming. Nothing streams until the tool call completes. Build the conversation flow around that gap and not against it.
4. Long context windows mislead. Cramming a prompt full is a bet that usually loses, because the more context there is, the more the model drifts at the edges. Supply only what's needed.
5. RAG adds power and upkeep. It brings a data pipeline that you now own, refresh and keep current, so weigh that cost honestly.
6. Live APIs beat static retrieval for real-time data. If freshness matters, querying the source directly outperforms a retrieval layer built on yesterday's snapshot.
7. Tool output needs a re-prompt. The model doesn't see a tool's result until it's added to the history and sent back. Forget that and the agent answers into thin air.
8. Prompts are architecture. Test and iterate on them the way you do code, since they hold the core logic of the agent.
9. Audio and transcripts diverge. What a caller heard and what got logged won't always agree, so QA can't rely on transcript accuracy alone.
10. Voice infrastructure isn't text infrastructure. Scaling lessons from text don't transfer. Voice has distinct latency profiles, connection behavior and failure modes, so start fresh.

What it costs

Doing it in-house costs about $8,500 for the simplest proof of concept, varying with volume and features, and $100,000 or more for an enterprise-grade system, plus three to six months of engineering before anything goes live. That's before compliance work and the 15 to 20% annual operating overhead that typically follows a production deployment.

| Tier                              | Upfront Cost | Timeline    | Key Features                                               |
|-----------------------------------|--------------|-------------|------------------------------------------------------------|
| MVP/Proof of Concept              | $8K-$30K     | 4-10 weeks  | Basic voice flows, single integrations, RAG memory         |
| Mid-Tier (CRM/multi-intent)       | $25K-$150K   | 2-6 months  | Contextual reasoning, telephony (Twilio), analytics        |
| Enterprise (autonomous, compliant)| $150K-$300K+ | 4-12 months | Multi-agent orchestration, multilingual, deep ERP/CRM,     |
|                                   |              |             | HITL governance                                            |

The team alone, covering AI engineers, backend developers, DevOps and QA, costs $150,000 or more per year, and that team is spending its time on infrastructure and not on the core business.

The minimum viable architecture

Six components come first, before anything else:

• Audio in. Twilio or WebRTC carries the caller's voice into the pipeline.
• Streaming STT. Deepgram Nova-3 transcribes live while the caller is still speaking.
• A model with a short prompt. GPT-5 or Claude Sonnet, keeping the system prompt under 500 tokens because bigger prompts slow responses.
• Streaming TTS. Cartesia or ElevenLabs begins playback before the whole answer is written.
• Logging. Keep transcripts and audio from every layer starting on day one.
• Fallback and handoff. Set a transfer rule for whatever the agent can't resolve.

Common questions

Is it possible to keep customer data away from third parties? Yes, through on-premise or hybrid hosting. Keep telephony and STT on your own infrastructure so raw audio stays contained, and let LLM inference run on managed cloud if compliance approves each layer.

Can voice AI handle calls and SMS in the same application? Yes. Both the telephony layer and the LLM orchestration layer work across channels, so the pipeline serving a phone call can route an SMS conversation through the same model and function-calling logic.

How do you fix poor intonation and pacing? Prosody and pacing come from the TTS model and not from the script. Cartesia and ElevenLabs both handle natural rise and fall well by default. If speech still sounds flat, the cure is usually a different voice model, not a different prompt.

How do you add voice AI to an existing phone system without changing numbers? Port the number to a SIP trunk through the telephony provider, then route inbound audio from that trunk into the STT layer. Callers dial the same number as before, and just the routing behind it is different.

How do you avoid vendor lock-in? Use the cascaded architecture. Every component runs independently, so Deepgram can be swapped for AssemblyAI or Cartesia for ElevenLabs without rebuilding the rest. Half-cascade and native speech-to-speech both give up that flexibility in exchange for lower latency.