Barge-in is a policy problem, not a model problem

The first time a voice agent talks over you, or keeps talking after you have started, the illusion of conversation breaks. It is tempting to treat that as a model problem and reach for a smarter model. In my experience it almost never is. Barge-in, turn-taking and "who speaks next" are policy: small, explicit rules that you can write down, test and tune. This post covers how two of my projects handle it: voiceAgent, a local assistant, and InterviewPracticeAI, an AI interviewer you practise with out loud.

Two stages, not one switch

The naive barge-in rule is "if the user makes a sound while the agent is speaking, stop". That fails both ways. A cough, an "mm-hm", or the agent's own voice leaking back into the mic all count as sound, and the agent stops mid-sentence for nothing.

voiceAgent uses two stages:

• Duck at 96 ms. After 96 ms of detected speech, TTS volume drops to 25%. It is cheap and reversible: if you were only saying "mm-hm", the volume comes back and nothing is lost.
• Cut at 320 ms. After 320 ms of sustained speech, playback stops, the queued audio is flushed and the model stream is cancelled.

In testing, a 0.3 s burst over the agent ducked the volume and was logged as a backchannel with no cut. A 2.8 s utterance over the agent was cut at 320 ms of speech and the new utterance answered.

user audio  ──▶ VAD ──▶ echo gate ──▶ barge-in accumulator
                                         │
                       ≥ 96 ms speech ───┼──▶ duck TTS to 25%
                      ≥ 320 ms speech ───┴──▶ stop TTS, cancel LLM stream,
                                               record what was actually heard

Speech is not continuous

The first version reset the barge-in counter to zero whenever a single chunk fell below the speech threshold. Real speech is never perfectly continuous, so sustained talking often failed to interrupt. The fix was a leaky bucket: a missed chunk costs one unit instead of wiping the count. The related bug was the opposite problem: the "backchannel ignored" note fired several times per utterance, with audible volume flicker, because ducking was released on the first stray miss. Now it only releases after about 190 ms of real silence.

Neither of those is a model change. Both were found by actually talking to the agent for a long session, then pinned down with deterministic tests against the real code.

Do not let the agent interrupt itself

With open speakers, the agent's own voice reaches the microphone. Without a guard, it barges in on itself. voiceAgent has an echo gate in front of the barge-in logic. In a synthetic test it flagged 23 of 26 echo chunks at 120 ms delay and a quarter of the level, while wrongly flagging 0 of 31 user-speech chunks and 0 of 26 user-over-echo chunks, at 0.7 ms per chunk. I am explicit in the report that real-microphone verification with open speakers still needs a human, and that is still true.

Remember what was not heard

When you cut the agent off, the conversation history should not claim it said the whole reply. voiceAgent records what had actually been spoken ("heard up to: ...") and drops the rest. In one test the barge-in landed 1.8 s into the reply, after "Hey, I hear you loud", with 12 words dropped. The next answer continued cleanly because the model knew where it had been cut off. If you interrupt before it has said anything, the history says exactly that rather than quoting an empty string.

An interviewer is mostly rules

InterviewPracticeAI takes the same idea further, because an interviewer has much stronger expectations about turn-taking than an assistant. The system is three parts sharing a typed protocol: a Next.js app, a Node WebSocket voice gateway that runs the turn-taking, model and TTS loop, and a Python "ear" doing VAD, end-of-turn detection and transcription on the GPU.

The interesting code is the interviewer policy in the gateway, and the test file reads like a list of interview etiquette:

• Never move on from a short answer. A two-word reply to "tell me about a project you led" gets a follow-up, not the next question.
• Advance only after the reaction was heard. The interviewer acknowledges your answer before moving on, and the next question waits until that acknowledgement has actually been played.
• Cap follow-ups. Probing is good. Five follow-ups on one question is an interrogation.
• Hold fragments and stitch long answers. A long answer can arrive as several segments with pauses in between. They are stitched into one answer rather than treated as several.
• "No no, you continue." If you accidentally cut the interviewer off and tell them to carry on, the question that was interrupted is replayed, with no model call.
• Re-read a question that was cut off before you could hear it.
• Survive failure. Model failures are handled, and after a reconnect the interview resumes at the first unanswered question.

None of these need a better LLM. Several of them, like "no no, you continue", are better handled with no LLM call at all: it is faster, cheaper and deterministic.

Testing policy with a scripted model

Because the policy is explicit, it can be tested without any real audio or real model. The gateway's interview session test drives the interviewer end to end with a scripted model and a scripted ear: feed in "the candidate said X, then paused, then said Y", and assert what the interviewer does next. There is a separate test for the WebSocket protocol against a stand-in web API, and one for the browser's reconnect and backoff behaviour. A live smoke test then streams real audio through a running gateway and checks transcript, reaction, TTS and completion.

That split is the whole argument of this post. The model generates the words. Everything about when those words are spoken, cut, replayed or skipped is ordinary state-machine code, and it deserves ordinary tests.

Takeaways

1. Use a cheap, reversible action (ducking) before an expensive, irreversible one (cutting).
2. Accumulate evidence with decay rather than requiring perfectly continuous speech.
3. Gate the agent's own echo before it reaches the barge-in logic.
4. Record what the user actually heard, not what the model generated.
5. Write turn-taking as explicit rules and test them with a scripted model and ear.

Code: github.com/nodeblackbox/voiceAgent and github.com/nodeblackbox/InterviewPracticeAI.