How to Build an AI Voice Agent (Latency, Turn-Taking, and Handoffs)

To build an AI voice agent you wire five pieces together: an audio transport, turn detection, speech to text, a model with a small tool set, and text to speech. That part takes an afternoon. The hard part is holding the whole round trip under about 800 milliseconds while never cutting the caller off mid-sentence. Latency and turn-taking are the product. Everything else is plumbing.

TL;DR

• Human turn transitions have a modal offset of 0 to 200 milliseconds across ten typologically diverse languages, per Stivers et al. in PNAS. That is the bar your caller's ear is calibrated to.
• Real phone calls tell a harsher story. Openbenchmarks' 2026 latency test measured roughly 432 calls per platform and found the fastest median time to first audio at 1,296 ms, with all five platforms above 1.2 seconds.
• Two stages own most of your budget: how long you wait to decide the caller is done, and the model's time to first token. Fix those before touching anything else.
• Cascaded (speech to text, then model, then text to speech) is still the production default because every stage is swappable and logs as text. Deepgram's breakdown covers the tradeoff honestly.
• All-in cost runs about $0.07 to $0.30 per minute once telephony and platform fees are counted. Silence bills the same as speech.

The five pieces, and which one you will actually fight

A voice agent is a loop that runs a few times per second.

Audio comes in over SIP (a real phone number) or WebRTC (a browser or app). A turn detector decides whether the caller has finished a thought or is just breathing. Speech to text converts the finished turn into a transcript, usually streaming partials the whole time. The model reads the transcript plus your instructions, decides whether to call a tool, and starts producing text. Text to speech converts that text to audio and streams it back before the sentence is finished.

Four of those five are commodities in 2026. You can swap providers in an afternoon and the call sounds roughly the same.

The fifth, turn detection, is where every bad voice agent you have ever spoken to went wrong. It ships with a default that is tuned for demos, and demos do not include a caller reading out a sixteen-digit account number with pauses in the middle.

Budget the latency before you write code

Write the budget down first, as a table, with a number next to each stage. Then measure against it. Without the table you will spend three days optimizing text to speech and discover it was never the problem.

Three rules for measuring it.

Measure over the real transport. A local WebSocket test will show you 400 ms and a PSTN call will show you 1,200. The phone network adds real time and your laptop does not simulate it.

Measure p95, not the mean. A benchmark of five commercial platforms found tail latencies running 1.24x to 1.48x the median, which means a chunk of turns land near or past two seconds even on the fast ones. Callers remember the two-second gap, not the average.

Measure from the caller's last word, not from your server's first log line. The clock starts when the human stops talking. Anything you do before that, including the pause you deliberately hold to be sure they finished, is inside the budget.

Turn detection is the whole product

Voice activity detection answers one question: is there speech energy right now? That is not the question you need answered. You need to know whether the caller is finished.

Those come apart constantly. "My number is four four two..." has a pause in it. So does "I want to cancel my... actually, wait." A detector watching energy alone hears silence and starts talking. Now you have interrupted a customer mid-account-number, and the recovery costs you fifteen seconds and most of their patience.

The fix is layered. Energy-threshold VAD stays as the cheap first pass. Above it sits a model that reads the partial transcript and the pitch contour and asks whether that was a complete thought. A trailing sentence with a rising pitch and no object is not a finished turn, no matter how long the silence is.

Barge-in is the mirror image. When the caller talks over the agent, the agent stops within about 100 milliseconds, drops the rest of the queued audio, and keeps whatever it already said in the transcript so it does not repeat itself. The failure mode here is a false barge-in: a cough, a TV, a coworker, and the agent goes silent mid-sentence for no reason. Guard it with a minimum sustained-voice duration of roughly 200 to 300 milliseconds before you accept an interruption, and a voice classifier rather than raw energy.

Track two numbers in production from day one: false interruption rate and missed barge-in rate. Both are per-call percentages, both are fixable, and neither shows up in any model benchmark.

Cascaded or speech to speech

Cascaded runs three models in sequence. Speech to speech runs one multimodal model that takes audio and returns audio with no transcript in between.

Speech to speech is faster and better at the human texture: the "mhm" while you talk, the tone shift when you sound annoyed, the natural overlap. It is also a black box. There is no text transcript in the middle to log, redact, evaluate, or hand to a compliance reviewer, and your provider choice narrows to a couple of vendors.

Cascaded costs you a few hundred milliseconds and buys back everything else. You can swap the model without touching the voice. You can run evals against the transcript. You can redact a card number before it reaches your logs. That is why it is still the default in enterprise deployments, and it is the right starting point unless you already know your use case lives or dies on tone.

The practical answer for most builders: cascaded first, then move one high-volume flow to speech to speech and A/B it on completion rate.

Give it three tools and one script

The model layer of a voice agent is not where the difficulty is, but it is where people over-build.

A voice agent should have a small tool set: look something up, write something down, transfer the call. Three to five tools. Every additional tool adds a decision the model makes out loud while the caller waits, and voice punishes deliberation in a way chat does not. If you are wiring the agent into real systems, the connection layer matters more than the count. The patterns in THE MCP BIG THREE are the same ones that apply here, just with a stricter latency ceiling.

The script matters more than the tools. Write the exact opening line. Write what the agent says when it does not understand, and make it different the second time. Write the transfer line. Voice has no scroll-back and no undo, so ambiguity that a chat user would tolerate becomes a caller talking over your agent trying to correct it.

One concrete constraint that saves entire calls: cap the response length. Two sentences, then stop and yield. A model that produces a well-organized four-sentence answer is producing a bad voice turn.

What it actually costs to run

Five line items: speech to text, model inference, text to speech, telephony, and the platform orchestration fee. Inworld's 2026 cost model and Retell's breakdown both land in the same range: roughly $0.07 to $0.30 per minute all-in, with most production deployments between $0.12 and $0.25.

Two things distort that number in a spreadsheet.

Silence bills at the same rate as speech. A 900 ms average gap across forty turns adds thirty-six seconds of billed dead air per call, so latency shows up on the invoice as well as in the experience.

Text to speech and the platform fee are usually the two biggest lines, not the model. Builders instinctively optimize model cost because that is the habit from chat. On a voice call the model is often the cheapest thing in the stack.

Model cost per resolved call instead. An agent that costs $0.22 a minute and resolves in three minutes beats one that costs $0.11 and takes nine, and only one of those is visible on a per-minute dashboard.

The handoff rule

Decide the transfer triggers before launch and write them into the config, not into a hope.

Three triggers cover almost everything. The caller asks for a person, and the agent transfers immediately without a retention attempt. The agent fails the same step twice, which means it did not understand the same thing two turns running. Or the conversation reaches money, cancellation, or anything legally binding.

Transfer with context. The human on the other end should receive the transcript and the identified intent, not an unexplained ringing line. A transfer that makes the caller repeat everything is worse than no agent at all, because now they have spent two minutes and still have to start over.

Set a hard turn cap per call and alert on it. If a call runs past that cap, something in your flow is broken and the transcript will tell you exactly where. That single alarm has found more real bugs in voice deployments than any eval suite.

If you want the build patterns behind agents like this as I ship them, they go out in the OpusJake newsletter most weeks.

The bottom line

A voice agent is not a chatbot with a microphone. The model is the easy part and the commodity part. What separates an agent people finish a call with from an agent people say "representative" to is a latency budget you actually measured, a turn detector that knows the difference between a pause and an ending, and a written rule for when to stop trying and get a human.

Build cascaded. Budget 800 milliseconds and instrument p95. Give it three tools and a two-sentence limit. Write the transfer triggers before you write the greeting. Then put it on a real phone line and call it yourself twenty times, because that is the only test that catches the thing that makes callers hang up.