Where the latency goes in a local voice agent

I wanted a voice assistant I could talk over, running entirely on my own machine, that answered quickly enough to feel like a conversation rather than a walkie-talkie. "Quickly enough" is vague, so the first thing I did on voiceAgent was measure every stage and write the numbers down.

All the figures below are from the project's benchmark report, measured on one machine: an RTX 4090, Windows 11, Python 3.11, native (no WSL), with a Blue Yeti mic. They are not universal. They are what this pipeline does on this box, and the point of the post is where the time goes, not the absolute values.

The pipeline

mic ─ Silero VAD ─┬─ EchoGate ─ StateMachine ─ barge-in
                  ├─ live partials (Parakeet re-decodes every 0.6 s for the UI)
                  └─ utterance ─ Parakeet ─ Brain (LangGraph + tools, streaming) ─ Kokoro ─ speaker

A spoken turn has four costs you pay in sequence: deciding you have finished talking, turning the audio into text, getting the first sentence out of the model, and turning that sentence into sound. Each one surprised me in a different way.

1. Speech recognition is the cheap part (once it is actually on the GPU)

Parakeet TDT 0.6B, run through onnx-asr, transcribes a 4.4 second utterance in 45 ms best / 79 ms median, and a 64 second monologue in 147 ms best / 169 ms median. That is a real-time factor of about 437 on the long clip.

Getting there took two fixes that are worth knowing about:

• Silent CPU fallback. With one cuDNN version the first convolution failed and onnxruntime quietly rebuilt the session on CPU. Every "CUDA" number in my first run was actually CPU (RTFx 13). I pinned cuDNN and the CUDA 12 runtimes and disabled the fallback so that a broken GPU setup is an error rather than a slow success.
• The decoder does not belong on the GPU. The transducer decoder runs once per 80 ms frame with tiny tensors, so on CUDA it is launch-bound at 0.5 to 0.7 ms per frame. The int8 decoder on CPU runs at 0.14 to 0.24 ms per frame. So the final setup is a hybrid: encoder on CUDA, decoder int8 on CPU.

mode	4.4 s clip	64 s clip
hybrid (CUDA encoder + int8 CPU decoder)	51 ms	171 ms
everything on CUDA	70 ms	461 ms
CPU int8	342 ms	4787 ms

2. Endpointing is where the real latency hides

Recognition takes tens of milliseconds. Deciding when to run it takes hundreds.

Silero VAD costs 0.6 ms per 32 ms chunk, so detecting silence is free. The problem is how long you wait after silence before calling the turn over. My natural thinking pauses sit at 0.3 to 0.6 seconds. With a silence-only endpointer, a 300 ms hangover split my 64 second monologue into 11 segments; it took 800 ms of hangover to keep it as one. And that hangover is paid on every single turn, including the short ones.

The fix was to stop using silence as the only signal. Smart Turn v3.2 is a small (8.7 MB) model that looks at the audio and estimates whether the utterance is complete. I cut the monologue at every point where a silence-only endpointer would have fired: it scored the mid-sentence cuts at around 0.01 and fired at the real sentence ends and the real end of the clip. It costs about 68 ms per call on the CPU.

With Smart Turn as the gate the hangover can drop to roughly 300 ms, which the report estimates saves about 300 ms per turn. Later live use showed it still ended turns on hedges like "um" and "yeah yeah so", so the final rule has three tiers: below 0.7 never end, 0.7 to 0.9 hold if the last live-caption word is a filler, 0.9 and above always end.

3. The model's first sentence, not its first token

For voice, time to first token is the wrong metric. You cannot speak half a word; you need a complete clause. So I measured time to the first complete sentence through LiteLLM, with streaming and a spoken-style prompt:

model	first sentence
groq/openai/gpt-oss-20b	269 to 328 ms
anthropic/claude-haiku-4-5	821 to 1309 ms
gemini/gemini-3.5-flash-lite	532 to 579 ms

Models that think before answering were in the seconds. In the full loop, Haiku 4.5 gave 1.7 to 1.9 s from the end of speech to first audio. The report breaks that down as 600 ms hangover + 125 ms ASR + 800 to 1000 ms to Haiku's first sentence + 180 ms of Kokoro. Swapping in Groq's gpt-oss-20b replaces the 800 to 1000 ms with about 300, which is why it is the default.

4. TTS: stream by sentence

Kokoro-82M renders a whole segment at once; it does not stream inside a sentence. So the streaming happens one level up. LLM tokens go through a sentence chunker, and a generator thread renders sentence N+1 while sentence N plays.

Per sentence (33 to 105 characters), Kokoro takes 133 to 182 ms at a real-time factor of 0.03 to 0.05. With a simulated LLM producing 25 words per second, time to first audio was 441 ms: the first clause closes at 0.28 s, Kokoro takes 150 to 200 ms, and then one 21 ms audio block. Over 22 seconds of playback there were zero gaps between sentences.

One detail I like: when the model calls a tool it usually says something like "Let me check." first. The chunker is flushed at the tool call, so that phrase is spoken during the tool round trip instead of after it.

Putting it together

Text in, using Groq and Kokoro, the agent went from turn start to first audio in 477 to 627 ms across three test turns. Add about 800 ms of speech-end-to-text from the live loop and a spoken turn lands at roughly 1.3 to 1.5 s, or about 0.9 s with Smart Turn gating.

What moved the needle, in order:

1. Choosing a model by time to first sentence, not benchmark scores.
2. Replacing a long silence hangover with a turn-completion model.
3. Sentence-level TTS pipelining, so speech starts before the reply is finished.
4. Making sure the GPU was actually being used, and then moving the decoder off it.

What is still slow

In the live loop, ASR runs 3 to 4 times slower (157 to 241 ms) than the same audio offline. The likely cause is the torch VAD loop and the CPU decoder competing for the GIL and cores in one process. The next step is to run Silero through onnxruntime or move ASR into its own process. Measuring each stage separately is what made that visible: the end-to-end number alone would just have said "a bit slow".

The code, benchmark scripts and the full report are on GitHub: github.com/nodeblackbox/voiceAgent.