Can a Local LLM Speak and Listen at the Same Time? | Full Duplex Voice AI

Summary

The video demonstrates a local, full-duplex voice assistant: a text-based LLM that can listen while it speaks, rather than waiting for the user to finish before responding. The creator’s main claim is that this behavior can be approximated with a conventional text model, without training a new speech-to-speech model or using voice activity detection (VAD).

How the System Works

• A streaming speech-to-text model transcribes microphone input.
• Every 600 milliseconds, the latest speech is sent to the LLM as a new message.
• The LLM either produces a few words for text-to-speech or outputs a “listen” marker to stay quiet.
• The loop continues while the user and model are both active. There is no VAD deciding when the user’s turn ends; the LLM makes that choice at each interval.

The creator compares this to a processor rapidly switching between tasks: the model still handles one turn at a time, but the turns are short enough to feel simultaneous. The approach is based on the Duplex Cascade paper. The video contrasts it with traditional voice assistants, which typically wait for silence and take turns like a walkie-talkie.

Model and Training Approach

The system uses a Qwen 3.6 35B mixture-of-experts model with a LoRA adapter. The creator trained only the model’s last four of 40 layers, reasoning that the earlier layers already interpret the incoming words while the later layers can learn whether to speak or keep listening.

To fit training on two 12 GB RTX 3060 GPUs, the creator cached the outputs of the first 36 layers and trained the remaining layers separately. The adapter has about 3.8 million parameters. Training examples were generated by another model, then divided into short turns and modified to include pauses, interruptions, and speech-recognition errors.

The creator reports roughly three hours of training. Inference runs through llama.cpp and takes about 240 milliseconds per turn, within the 600 ms interval.

Experiments and Lessons

• Prompting alone helped but was unreliable. The model sometimes stayed silent too often, and repeated “listening” responses in its conversation history made that behavior increasingly likely.
• Fine-tuning required balancing different behaviors. Too many silent examples made the model reluctant to answer; too many brief acknowledgments led to excessive “yeah” or “mhm” interjections. The creator also found that longer training conversations improved responses during longer calls.
• A decoding rule addressed prematurely unfinished sentences. The model sometimes chose the “listen” token when it should continue speaking. The creator monitored that token’s probability: at 30% or higher, the model listened; below that, the token was removed so the model had to continue. The creator reports only two unfinished sentences in roughly 500 test sentences after applying the rule.

Reported Results and Limitations

The creator says the fine-tuned model learned to wait more appropriately and could interrupt a potentially dangerous action in a test, though its safety intervention remained inconsistent across tests. Other limitations included a roughly three-second delay before the first response, occasional poorly timed interjections, and cases where it failed to speak when it should.

The video compares this approach with systems such as Moshi and PersonaPlex, described as smaller open models requiring more GPU memory, and notes that those purpose-built systems start responding faster. It also argues that retaining a larger general-purpose model may preserve stronger language understanding. The creator cites Voicebench scores of around 30 for the compared 7B duplex models versus around 70 for a typical speech-to-text system paired with a 7B LLM. These comparisons and performance figures are presented by the video’s narrator.

Guide and Tutorial Elements

The video provides a practical overview of:

1. Building a 600 ms speech-to-text → LLM → text-to-speech loop without VAD.
2. Trying a system prompt before fine-tuning.
3. Training a LoRA on only the final layers using cached outputs from earlier layers.
4. Balancing training examples and adding a probability threshold to control when the model listens.

The creator says the LoRA adapter and a small client for trying it with llama.cpp will be made available.

Main Speakers and Sources

• Main speaker: The video’s creator and narrator, from Codacus.
• Referenced sources and systems: Thinking Machines Lab’s interactive-model demo; OpenAI GPT Live; the Duplex Cascade paper; Qwen 3.6 35B and llama.cpp; Moshi and PersonaPlex; Voicebench.