Speech To Speech: Build voice agents with open-source models
A low-latency voice agent built from open models that runs entirely on your computer. One command starts a conversation: no API key, no cloud service.
The default pipeline is Silero VAD → Parakeet TDT → Qwen3.5-4B → Qwen3-TTS, and every stage is swappable. The server speaks the OpenAI Realtime protocol over WebSocket and WebRTC, so apps built for OpenAI's Realtime API can point at your own machine instead.
This pipeline runs in production as the conversation backend for thousands of Reachy Mini robots.
Quickstart
Use CPython 3.11–3.14 (3.11 recommended):
python3 -m venv .venv
source .venv/bin/activate
pip install speech-to-speech
speech-to-speech local
Then talk: speak, pause, and the assistant answers through your speakers. Stop with Ctrl+C. The first run downloads the models, so give it a few minutes. On speakers, add --echo-cancellation so the assistant does not hear itself and you can still interrupt it by voice; headphones don't need it. See echo cancellation for requirements and limits.
By default, local and talk show final user transcripts and streaming assistant
replies once, hiding backend transcript copies, routine logs, tool details, and protocol
events. User messages are yellow and assistant replies are green in a color-capable
terminal. Startup shows Loading models…, followed by Ready — start speaking. once
the audio devices are running and the session is configured. Routine model-loading
output is hidden; startup failures show their diagnostics. Runtime warnings and errors
remain visible.
For diagnostic output, add --verbose:
speech-to-speech local --verbose
Use --verbose --log_level debug for more detailed local logs. Raw event output
(local --local_audio_print_json or talk --print-json) also enables diagnostic output.
On Ubuntu, first install the audio libraries with sudo apt-get install libportaudio2 libsndfile1. The default Linux Qwen3-TTS wheel targets CUDA 12.8 and Ubuntu 24.04; see the CUDA note for other systems. On Windows, activate the environment with .venv\Scripts\Activate.ps1.
The default installation has the following CI coverage:
| Platform | Supported CPython versions | Checks |
|---|
| Linux | 3.11, 3.12, 3.13, 3.14 | Full test suite and installed package smoke test |
| Apple Silicon macOS | 3.11, 3.12, 3.13, 3.14 | Installed package smoke test |
Optional model backends can have narrower Python requirements. Windows is not
covered by this compatibility matrix. Python 3.10 is no longer supported following
its end of life on October 1, 2026.
Kokoro is optional on every platform. On macOS, its Misaki dependency requires
Python <3.13, so use Python 3.11 or 3.12 with speech-to-speech[kokoro].
The default Qwen3-TTS backend does not require Misaki. Python 3.15 is not yet
supported on any platform: ONNX Runtime lacks CPython 3.15 wheels, and the pinned macOS PyTorch
release also lacks them. Newer Python versions will be added after the default
dependencies and CI checks pass; the package's requires-python lower bound does
not promise compatibility with future Python releases.
The command picks local models for your platform:
| Platform | LLM | Plan for |
|---|
| Apple Silicon Mac | Qwen3.5-4B 4-bit via MLX LM | 16 GB of unified memory |
| Linux / Windows with NVIDIA | Qwen3.5-4B via Transformers | 24 GB of VRAM |
| Other machines | Qwen3.5-4B via Transformers on CPU | Slow replies |
Speech recognition and synthesis run locally too. After the first run, everything works offline. To use another model, a llama.cpp server, or a hosted provider, see LLM backends.
Connect your own app
local runs a server and a microphone client in one process. To run only the server:
speech-to-speech serve
It listens at ws://127.0.0.1:8765/v1/realtime. Connect with:
- the packaged audio client:
speech-to-speech talk --url ws://127.0.0.1:8765/v1/realtime
- the browser demo
- any client using the core OpenAI Realtime events; the OpenAI Agents SDK is tested over both transports (see Realtime API)
- LiveKit Agents, as the agent's realtime model
Index
How it works
The pipeline is a cascade of four components, each running in its own thread and connected by queues:
- Voice Activity Detection (VAD): Silero VAD detects speech boundaries and turn-taking.
- Speech to Text (STT): transcribes the user's turn, with optional live partial transcripts.
- Language Model (LLM): generates the response, streaming text and tool calls.
- Text to Speech (TTS): synthesizes audio and streams it back to the client.
This is the beginning of the README. Read the rest at the original. Read the full README ↗
