Where the second goes: a latency budget for AI voice

“Sub-second response” is a marketing sentence until you write down where the time
goes. On a live call the budget is spent in a fixed order, and each stage is a
different engineering problem.

Audio has to arrive. Network transit and jitter buffering cost tens of milliseconds
before anything has been recognised, and that cost is set by the route the call
takes, not by any model choice. Then speech recognition has to decide the caller has
finished — endpointing — which is the single most misunderstood stage. A system
waiting for a confident silence threshold can burn several hundred milliseconds
doing nothing at all. Shorten it and you start cutting people off mid-sentence,
which is worse than being slow.

Then the language model runs, and this is the part everyone tries to optimise first.
It is rarely where the second goes. Finally the reply is synthesised, and the first
audio frame is what matters — a system that waits for the whole sentence before
playing anything adds the full synthesis time to every turn, while one that streams
begins speaking as soon as the opening words exist.

The useful discipline is to measure each stage separately, per call, and look at the
distribution rather than the mean. A median of 900ms with a tail at four seconds is
a worse product than a flat 1.2 seconds, because callers remember the four-second
turn and nothing else. Averages hide exactly the calls that lose you customers.

Two practical notes. First, the biggest wins usually come from the stages nobody
demos: endpointing and streaming the first audio frame. Second, some latency is
worth paying for — a lookup against a live system that returns the caller’s actual
order status is worth several hundred milliseconds, because the alternative is a
fast answer that is useless. What is never worth it is latency the caller cannot
attribute to anything: silence with no acknowledgement.

If you take one thing away: instrument each stage now, before tuning anything. You
cannot optimise a budget you have not itemised.