Phony.ai

Voice AI latency figures are everywhere — but without a shared measurement definition, they're nearly impossible to compare. This episode breaks down the four-leg pipeline that every caller silently waits through, and explains why shaving milliseconds off the model is often the wrong place to look.

Show Notes

Sub-second voice AI sounds like a model achievement. It isn't. This episode of Phony.ai pulls apart the full end-to-end pipeline that determines how long a real caller actually waits — from the moment they stop speaking to the moment they hear the first syllable of a response — and makes the case that almost every published latency figure is measuring something different, and usually something narrower. The conversation is grounded in the Phony.ai deep-dive on voice response architecture, which traces how the team's production numbers moved from 3.5–4.5 seconds to 2.2–3.2 seconds in a single week — without changing the model.

Here's what the episode covers:

  • Why latency comparisons are almost meaningless. Vendors quote response times without defining the measurement window, making cross-vendor benchmarks impossible to interpret with confidence.
  • The four-leg pipeline every caller pays for. Endpointing, transcription, model inference, and speech synthesis are sequential — and three of those four legs barely move regardless of how fast the model is.
  • The honest floor for endpointing. Pushing the silence threshold below ~300 milliseconds causes the system to clip callers mid-sentence; the clipped fragment transcribes to nothing, and the caller experiences being ignored — a worse outcome than a longer wait.
  • Transcription as the current bottleneck. On the web path, transcription accounts for roughly 830 milliseconds and is the single largest contributor — a constraint shaped by the fact that batch transcription can't begin until speech ends.
  • The model is the one leg that's actually free to change. A small, fast voice-tuned model comes in around 690 ms to first token; a frontier model runs closer to 2.5 seconds — a factor of ~3.5× on the one variable that isn't constrained by physics or architecture.
  • Two architectural moves that actually shift the floor. Speculative execution (generating against a predicted complete utterance and discarding if the caller continues) and speech-to-speech models that collapse the pipeline entirely are the levers that matter — not prompt caching or reasoning-mode toggles, both of which the team investigated and found innocent.
  • Geography is a variable, not a footnote. Network distance between a host and a provider can swing results by 150 milliseconds. Any latency figure without a region attached is really a figure about one specific data centre.

The episode closes with a practical challenge: the next time a vendor quotes a sub-second response time, ask exactly what was measured between. Even with a hypothetical zero-latency model, endpointing, transcription, and synthesis still sit in the chain — and they don't disappear with a faster GPU. For more on how Phony.ai approaches product decisions around caller experience, check out the earlier episode Why There's No List Upload — And Why That's the Whole Point.

Phony.ai

What is Phony.ai?

AI phone and voice agents, explained through the constraints that actually decide whether one works: end-to-end latency and where it comes from, interruption and barge-in handling, telephony plumbing and call control, transfer design, and the disclosure and recording rules around automated calls.

Each episode takes one design decision and works through it concretely — provider-neutral, comparing approaches rather than selling one. Written for teams evaluating, buying or building voice AI who need to know what breaks before it breaks in production. Five or six minutes an episode.

Topics include end-to-end latency and where it comes from, barge-in and interruption handling, telephony and call control, transfer and escalation design, prompt and turn design, evaluation and call review, and disclosure, consent and recording rules.

Produced by Phony.ai, provider-neutral AI phone and voice agents. Full details, services and further reading at https://phony.ai