Short answer: A voice agent sounds natural on the phone when it does three things well: it stops speaking quickly when the caller interrupts (barge-in), it decides correctly when the caller has finished a turn (endpointing), and it answers fast enough that the pause does not feel like a dropped line. All three depend on settings and testing, not only on the platform. Test them with real phone audio, real accents and real background noise before the agent takes a single customer call.
We build voice assistants for small businesses and e-commerce brands, and turn-taking is where most of the tuning time goes. A correct answer delivered while talking over the caller still feels broken.
Why phone conversations are hard for AI
People do not take turns politely. They interrupt, say "mm-hm" while you talk, pause in the middle of a sentence to find an order number, and start again. A phone line adds its own problems: narrowband audio, compression, background noise, and a small delay in each direction.
A voice agent usually runs a pipeline of separate parts, or a single speech-to-speech model that combines them:
- Voice activity detection (VAD) decides whether the sound on the line is speech.
- Speech-to-text (STT) turns that speech into words, often streaming partial results.
- Turn detection or endpointing decides the caller has finished.
- The language model decides what to say and which tools to call (look up an order, check a calendar).
- Text-to-speech (TTS) speaks the reply, ideally streaming the first words before the full reply is ready.
Each step adds delay, and each one can make a turn-taking mistake.
The three things that make or break a call
| Problem | What the caller experiences | Main cause | What to tune |
|---|---|---|---|
| Talking over the caller | Agent keeps speaking after "no, wait" | Barge-in disabled or too insensitive | Interruption sensitivity, minimum words before stopping |
| Cutting the caller off | Agent replies while caller is mid-sentence or reading a number | Endpointing too aggressive | Silence threshold, semantic turn detection, longer wait after digits |
| Long, awkward pauses | Caller says "hello?" | Slow model, slow tools, no streaming | Smaller or faster model, streaming TTS, filler phrases, faster tool calls |
| Stopping for "mm-hm" | Agent stops for every acknowledgement | Barge-in too sensitive | Ignore short backchannel words, require a minimum duration |
| Losing the thread after an interruption | Agent repeats itself or forgets what it said | Context not updated with what was actually spoken | Truncate the agent's last turn to what the caller heard |
Barge-in: stopping when the caller talks
When the caller starts speaking, the agent should stop its audio quickly and listen. The fine print matters:
- Do not stop for every sound. Coughs, a door, a "yeah" or "okay" should usually not stop the agent. Most platforms let you set a minimum speech duration or a number of words before an interruption counts.
- Record what the caller actually heard. If the agent was cut off halfway through a sentence, its memory of the conversation should contain only the half that was spoken. Otherwise it may assume the caller heard information they never heard.
- Respond to the interruption, not the script. "Sorry, go ahead" and then answering the new question feels natural. Restarting the old sentence does not.
Endpointing: knowing when the caller has finished
A simple endpointer waits for a fixed amount of silence. Too short, and the agent jumps in while the caller is thinking. Too long, and every reply feels slow. Better approaches combine silence with meaning: if the words so far form a complete question, reply sooner; if the caller is reading out a phone number, an order number or an address, wait longer. Several voice platforms now offer this kind of "semantic" or model-based turn detection; check what yours supports and test it with your own scripts.
Latency: the pause before the answer
On a phone call, every part of the pipeline adds delay, and the network adds more in each direction. The ITU recommendation for telephone networks, ITU-T G.114, treats one-way transmission delay up to 150 milliseconds as acceptable for most applications, and that is before your agent starts thinking. The practical lessons:
- Stream everything you can: partial transcripts in, first words of speech out.
- Keep tool calls fast. An order lookup that takes several seconds needs a short spoken bridge ("Let me check that order for you") so the silence has a reason.
- Choose the model for the job. The largest model is not always right for a phone line where speed matters more than depth.
- Host near your callers and your telephony provider where the platform allows it.
Speech-to-text best practices for customer calls
The transcript is the foundation. If the words are wrong, every later step is wrong.
- Use phone audio for testing. A model that is accurate on studio audio can struggle with narrowband, compressed phone audio. Test with recordings made over real calls (with consent and in line with your privacy notice).
- Add custom vocabulary. Product names, street names, brand terms and staff names are where generic models fail. Most STT services accept keyword boosting or a custom vocabulary list.
- Confirm critical values. Read back order numbers, dates, phone numbers and amounts. Do not let the agent act on a number it heard once.
- Handle accents and code-switching. If your callers mix languages, test that mix specifically. Our guide to multilingual voice AI covers language detection.
- Keep the transcript. A stored transcript, with a recording where lawful, is how you find and fix the calls that went wrong. Tell callers if calls are recorded, and keep the data only as long as you need it.
Choosing a platform for production
There is no single best platform; the right choice depends on your telephony, languages and how much control you need. Our comparison of ElevenLabs, Vapi and Retell goes deeper. When you evaluate any platform for production, ask:
- Can I tune barge-in sensitivity and endpointing, and see the settings in one place?
- Does it support semantic turn detection, or only a silence timer?
- Can I bring my own phone numbers and carrier, or must I use theirs?
- Does it stream STT and TTS, and can I choose the models?
- Are full transcripts, recordings and per-turn timings available for every call?
- Can it transfer a call to a person with the context attached?
- Where is call data stored and processed, and is there a processor agreement?
A test checklist before real callers
Run each scenario at least a few times, on a real phone line, with more than one voice:
- Caller interrupts the greeting with a question.
- Caller says "mm-hm" and "okay" while the agent explains something.
- Caller reads out a long order number with pauses.
- Caller changes their mind mid-sentence.
- Caller is in a noisy room or a car.
- Caller has a strong regional accent, or switches language.
- Caller asks something the agent cannot answer and needs a person.
- A tool call is slow or fails.
Score each call on three questions: did the agent stop when it should, did it wait when it should, and did any pause feel like a dropped line? Listening to recordings is still the most reliable test. Automated voice testing tools can generate calls at scale and are useful for regression checks after each change, but they do not replace a person listening.
If you want to hear how a grounded assistant handles turn-taking, try the voice AI demo, or read how we approach voice AI for restaurants, where interruptions and background noise are constant. The voice AI page describes what we build.
Frequently asked questions
What is barge-in in a voice AI agent? Barge-in means the caller can interrupt while the agent is speaking, and the agent stops and listens. Good barge-in ignores short sounds like a cough or a quick "mm-hm" and records only the part of its own sentence the caller actually heard.
Why does my voice agent cut callers off? Usually because endpointing is too aggressive: it treats a short pause as the end of the turn. Increase the silence threshold, use semantic turn detection if your platform has it, and allow longer pauses when the caller is reading numbers or addresses.
How fast should a voice agent respond on the phone? Fast enough that the pause does not feel like a dropped line. The network already adds delay, as ITU-T G.114 notes, so stream speech in and out, keep tool calls quick and use a short spoken bridge when a lookup takes time.
How do I test a voice agent before launch? Call it on a real phone line with several voices, accents and background noise, and run scripted scenarios: interruptions, long numbers, changed minds, unanswerable questions and failed tool calls. Listen to the recordings and fix one setting at a time.
Do automated voice agent evaluation tools replace human testing? No. They are useful for running many calls and catching regressions after a change. A person listening to recordings still catches tone, awkward pauses and misunderstandings that automated scores miss.