audio.cpp

audio.cpp is a high-performance C++ audio inference framework built on top of ggml, designed to make modern local audio models practical, portable, and fast.

Tired of juggling a dozen Conda environments, hundreds of Python packages, and dependency conflicts just to try a few audio models? audio.cpp gives those paths a shared native runtime instead. Runs on Windows, Linux, and macOS, with support for NVIDIA, AMD, Apple Silicon, and CPU-only machines.

Lost in a sea of audio model names, papers, and codebases? Explore 100+ audio models in the Audio Model Architecture Atlas to see how they're built, what they share, and where they differ, whether you're learning or building your own.

[!TIP]

v0.9.3 highlights: LoRA support across 11 models, with adapter loading, strength controls, Studio integration, and a shared tool for merging adapters into GGUF packages. See the LoRA guide for supported models and formats.

Performance optimizations and performance modes: Faster execution across speech and music models, with perf_mode presets to select model-specific optimizations. Tested CUDA presets reduce RTF by 27% for Qwen3-TTS streaming, 25–26% for BreezeTTS2, 16% for VoxCPM2, 15–47% for YuE2, 59–62% for ACE-Step, and 33% for MiniMax Music3 versus the paths before these optimizations. Some models also benefit from further reductions in peak VRAM usage.

Many performance optimizations are disabled by default for broader compatibility. Not happy with performance? Enable the cuda_perf preset or create your own preset for your hardware!

New TTS app in the WebUI: Turn scripts into narration in a dedicated workspace. Import text, split it into passages, and assign each passage a reusable speaker with its own model, voice, and generation settings. Generate or refine passages individually, compare takes, edit waveforms and pauses, then combine, enhance, and export your finished audio. Projects and speaker presets are saved for reuse.

AMD support and HIP test releases: Thanks to AMD for providing audio.cpp with access to AMD DevLab! We now offer HIP release builds for testing on AMD GPUs.

[!IMPORTANT]

• 130+ audio model families and 200+ variants, running locally. TTS, ASR, music generation, voice conversion, separation, and more through a shared CLI and server API.

• Built-in WebUI: Generate and clone voices in Studio, compare models side by side in Arena, manage models and voice/speaker libraries, and create narration with passage editing, multiple speakers, and audio assembly in the TTS app.

• Performance: Selected CUDA TTS paths run 1.8–8× faster than Python; tested Q8 packages offer up to 1.53× speedup and 37% lower peak VRAM versus 16-bit weights (report). VibeVoice 1.5B reaches 5.15× real time, and Supertonic 3 generates 10 hours of audio in 3 minutes on RTX 5090 (demo). Nemotron 3.5 ASR matches WER at roughly ¼ the wall time in the TranscrIA benchmark.

• Integrations and deployment: audio.cpp powers speech, music, and speech-to-text in Unsloth Studio. Deploy Fun-ASR-Nano and SenseVoice through the FunASR platform!

It is built for real end-to-end execution rather than one-off model demos: the same runtime powers TTS, voice cloning, voice conversion, ASR, diarization, VAD, source separation, alignment, codec-style models, and higher-level workflows through a common framework surface.

Highlights:

• Parity. Strong parity tooling against Python reference paths.
• Performance. Performance-focused execution, reusable sessions, and batch-style offline inference. Optimized for CUDA.
• Portability. A portable native stack centered on ggml, with CUDA, HIP/ROCm, Vulkan, Metal, and CPU backends behind shared CLI and server entry points instead of Python-only deployment paths.
• Pipelines. Experimental JSON pipeline support for higher-level multi-step workflows.
• Audio Utilities. Built-in denoise, enhancement, resampling, and STFT/ISTFT utilities for real production-style task paths. Model-based utilities need separately downloaded weights; see downloads and usage.

audio.cpp would not be moving this quickly without generous contributors bringing in real fixes, new capabilities, and careful polish. See CONTRIBUTING.md for how to contribute and for a shout-out to the people already helping shape the project.

[!TIP] Contribution focus: the most helpful contributions right now are improvements to the UI, API server, and pipeline/workflow subsystems. These areas make the existing model surface easier to use, serve, compose, and validate. See CONTRIBUTING.md for more details.

New model PRs: before starting a new model port, please check the supported model table because several families are already implemented or under testing and read the New Model PRs section in CONTRIBUTING.md. New ports should start under the community model source tree, where review is lighter than core models but still needs reproducible validation.

News

[!IMPORTANT] Release v0.9.3: New models include Fish Audio Drama3 and Drama3-small, Ming 3.0 TTS, Ming 3.0 Audio Tiny, Coqui XTTS v2, Plapre 3, Gradium Phonon, and Earshot VAD, plus experimental LatentSync 1.6 lip synchronization. Existing families gain Irodori-TTS MeanFlow and multilingual Granite TurboCTC support.

New features include LoRA support across 11 models, Studio adapter controls and GGUF merging, a TTS workspace with reusable speakers, Ming3-TTS streaming, Kokoro voice blending, and built-in recipes for voice cloning and cross-language dubbing. Performance optimizations and perf_mode presets improve speech and music generation, with further memory reductions for selected models. Hardware support expands with HIP test releases, more ARM NEON acceleration, and additional Rockchip NPU paths and Docker images. Thanks to everyone contributing code, testing, feedback, and hardware access!

Release v0.9.2: New models include Voxtral TTS, TF-GridNet, MossFormer2, Hviske v6, FireRed VAD, AST, CED, and wake-word detection models. This release expands TTS streaming, adds multi-turn LFM2.5-Audio conversations, and improves model performance and compatibility. The WebUI gains a shared Voice Library and layout improvements. Downloadable C API libraries, Windows Vulkan portable builds, and packaged server configuration examples make integration easier. Thanks to everyone contributing code, testing, and feedback!

2026-09-30 - Release v0.9.0: 100+ audio models, an enhancement and denoising WebUI tab, speaker-tagged ASR, and performance and memory improvements. Explore the Architecture Atlas. Thanks to our community!

2026-06-25 to 2026-09-23 (releases 0.1 to 0.8): audio.cpp grew to 80+ model families and 120+ variants across TTS, ASR, music generation, voice conversion, separation, and speech analysis. Major additions included the native WebUI and Arena, GGUF packaging, LoRA adapters, streaming, and CUDA/Metal performance improvements, alongside Vulkan and experimental HIP support. The C ABI, Colab integration, MP3/HTTPS frontends, and models such as YuE2 and SheetSage2 expanded integration and audio-to-score workflows.

Supported Models

Task tags: TTS text to speech, Clone voice cloning, VC voice conversion, S2S speech-to-speech, ASR speech recognition, Classify audio classification, Wake wake-word detection, Align forced alignment, VAD voice activity detection, Diar speaker diarization, Codec audio codec, Sep source separation, MIDI audio-to-symbolic MIDI/events, Music music/song generation, SFX sound effects, Video video generation, Edit audio/music editing, Design voice design, Dialogue multi-speaker dialogue TTS, Ctrl TTS/clone voice control such as emotion, style, instruction, caption, or non-verbal tag control.

Runtime tags summarize the supported loading paths. GGUF package precision varies by model and release; check the audio.cpp GGUF repo or docs/gguf.md for the exact package list. Bundled means the tiny runtime asset ships under assets/framework/models and needs no separate model download. Stream means the family exposes a streaming server/session path. LoRA means adapter loading is supported in audio.cpp; see the LoRA guide for supported formats and usage.

Model weights keep the license of their original release, which is separate from audio.cpp's own license. docs/model_licenses.md lists that license per family, and whether it allows commercial use.

Speech Generation And Conversation

Family	Task	Lang	Variants	Runtime
breeze_tts	TTS, Clone, Design, Ctrl	zh, en	BreezeTTS 2 instruction-conditioned TTS and prompt-audio voice cloning	GGUF BF16/Q8/Q4_0, Stream
chatterbox	TTS, Clone, VC	ar, da, de, el, en, es, fi, fr, hi, it, ko, ms, nl, no, pl, pt, sv, sw, tr	Chatterbox with 0.5B backbone	GGUF 16/Q8
confucius4_tts	TTS, Clone	zh, en, ja, ko, de, fr, es, id, it, th, pt, ru, ms, vi	Confucius4-TTS multilingual voice cloning	GGUF F32, Stream
cosyvoice3	TTS, Clone	zh, en, ja, ko, de, es, fr, it, ru, yue	Fun-CosyVoice3 zero-shot, cross-lingual, and instruction-conditioned TTS	GGUF F32/Q8, LoRA
dots_tts	TTS, Clone, Edit, Ctrl	multilingual	DotTTS SOARDotTTS MeanFlowDotTTS Edit	GGUF 16/Q8, Stream
dramabox	TTS, Clone	en	DramaBox expressive TTS and voice cloning	GGUF Q8
fish_audio	TTS, Clone, Ctrl	auto, en, zh	Fish Audio S2 Pro	GGUF 16/Q8, Stream, LoRA
firered_audio	ASR, TTS, Clone, Design, Ctrl	zh, en	FireRedAudio multimodal speech/audio model with ASR, understanding, cloning, design, and edit paths	GGUF original/Q8
fireredtts3	TTS, Clone, Design, Ctrl	24 langs + 21 zh dialects	FireRedTTS3 BaseFireRedTTS3 Instruct/Voicedesign	GGUF original/Q8
gradium_phonon	TTS	en	Gradium Phonon 31m English	GGUF Q8, Safetensors, Stream
higgs_audio_tts	TTS, Clone, Ctrl	auto	Higgs Audio v3 TTS 4B	GGUF 16/Q8, Stream
index_tts2	TTS, Clone, Ctrl	zh, en, ja, es, ar	IndexTTS-2IndexTTS-2.5	GGUF 16/Q8

Speech Recognition And Analysis

Family	Task	Lang	Variants	Runtime
ast_audioset	Classify	language agnostic	MIT AST AudioSet, 527 classes	GGUF F32
ced	Classify	language agnostic	CED TinyCED MiniCED SmallCED Base527 AudioSet classes	GGUF F32
audio_flamingo	ASR, Audio understanding	en	Audio Flamingo 3Audio Flamingo NextMusic Flamingo 2601SpotSound (LoRA temporal grounding)	GGUF BF16/Q8/Q4, LoRA
canary_asr	ASR, Translate	en, de, es, fr	Canary 180M Flash	GGUF F32/Q8
citrinet_asr	ASR	en	Citrinet-256	GGUF Q8
cohere_asr	ASR	en, fr, de, es, it, pt, nl, pl, el, ar, ja, zh, vi, ko	Cohere Transcribe 03-2026	GGUF BF16/Q8/Q4_0
crisperwhisper	ASR, Align	99 language codes	CrisperWhisper 2.0 Large	GGUF BF16/Q8, Stream
gigaam_asr	ASR	ru, kk, ky, uz, en	GigaAM v3 CTCGigaAM v3 RNN-TGigaAM v3 E2E CTCGigaAM v3 E2E RNN-TGigaAM Multilingual CTCGigaAM Multilingual Large CTC	GGUF F16/F32
fun_asr_nano	ASR	auto, zh, en, ja	Fun-ASR-Nano-2512	GGUF 16/Q8
higgs_audio_stt	ASR	en	Higgs Audio v3 STT	GGUF 16/Q8, Stream
hviske_asr	ASR	da	Hviske v5.3	GGUF Q8
hviske_asr_v6	ASR	da	Hviske v6	GGUF BF16 / Q8
index_echo	ASR, Translate	zh → en, es, ja	Index-Echo-S2TT 2BIndex-Echo-S2TT 9B	GGUF original/Q8_0/Q4_K
marblenet_vad	VAD

Audio Conversion And Processing

Family	Task	Lang	Variants	Runtime
apollo	S2S	lang agnostic	Apollo music restoration	GGUF F32
audiosr	S2S	lang agnostic	AudioSR Basic audio super-resolution package	GGUF F32
bs_roformer	Sep	lang agnostic	BS-RoFormer vocal separation checkpoints	GGUF Q8
builtin_audio_utils	S2S	lang agnostic	RNNoiseDeepFilterNet2ZipEnhancerGTCRN StreamingGTCRN DNS3GTCRN VCTKFlashSR	Safetensors
controlfoley	SFX	auto	ControlFoley 44 kHz multimodal Foley generation from text, video, and reference audio conditioning	GGUF F32/Q8
htdemucs	Sep	lang agnostic	HTDemucsHTDemucs_ftHTDemucs_6stems	GGUF 16/Q8
meanvc2	VC	lang agnostic	MeanVC2 120 ms/40 ms zero-shot voice conversion	GGUF F32/Q4, Stream
mel_band_roformer	Sep	lang agnostic	Mel-Band RoFormer MLX vocal separation variants	GGUF 16/Q8
miocodec	Codec, VC	lang agnostic	MioCodec v2, 25 Hz, 44.1 kHz	GGUF 16/Q8
mossformer2	Sep	lang agnostic	MossFormer2 SS 16K two-speaker separation	GGUF F32
muscriptor	MIDI	music	MuScriptor Small audio-to-symbolic transcription	GGUF F32, Stream
rvc	VC	lang agnostic	RVC F16 GGUF with packaged v1/v2 voices and optional retrieval blending	GGUF 16
sam_audio	S2S	lang agnostic	SAM Audio SmallSAM Audio BaseSAM Audio Large	GGUF F32/BF16/Q8
seed_vc

Music, Media, And Editing

Family	Task	Lang	Variants	Runtime
ace_step	Music, Edit	50+ langs	ACE-Step 1.5 TurboACE-Step 1.5 BaseACE-Step 1.5 XL TurboACE-Step 1.5 XL SFT	GGUF 16, LoRA
heartmula	Music	zh, en, ja, ko, es	HeartMuLa-oss-3B with HeartCodec-oss	GGUF 16/Q8
magenta_realtime_2	Music	music	Magenta RealTime 2 Small (230M)Magenta RealTime 2 Base (2.4B)	GGUF F16/Q8_0, Stream
midashenglm_gen	Music, SFX	auto	MiDashengLM-Gen structured-prompt generation for speech, music, sound effects, and ambience	GGUF F32/Q8
minimax_h3	Video, Music, TTS/Dialogue	auto	MiniMax-H3 Q4_K with optional INT8 ConvRot DiT	GGUF Q4/INT8
minimax_music3	Music	auto	MiniMax Music 3 text-to-music generation with lyrics conditioning	GGUF Q4/Q8
stable_audio	Music, SFX, Edit	en	Stable Audio 3 Small MusicStable Audio 3 Small SFXStable Audio 3 Medium	GGUF 16/Q8
vevo2	TTS, Music, VC, Edit	en, zh	Vevo2 with Qwen2.5-0.5B AR model	GGUF 16
yue2	Music	en	YuE2-3B lyrics-conditioned song generation with optional ABC score planning and conditioning	GGUF BF16/Q8/Q4, LoRA

Some model families in the supported table started as outside contributions before being promoted into the core release surface. Thanks to Mirek @mirek190 for BS-RoFormer, @justinjohn0306 for MOSS-TTS-Local, and @LauraGPT from the official FunASR team for Fun-ASR-Nano.

Community Models

Community model ports live under community_models to make the ownership boundary clear while keeping them available through the normal audio.cpp CLI and server paths. Some community-contributed models graduate into the core model tree when they become part of the main release surface. Huge thanks to the contributors who bring these models in, test them, and keep pushing the framework into new territory. See CONTRIBUTING.md for community-model expectations.

Family	Task	Lang	Runtime	Contributor	What They Added
audio8_asr	ASR	en, zh, yue, ja, ko, fr, de	GGUF Q8, Safetensors	@gqf2008	Audio8-ASR-0.1B compact multilingual autoregressive ASR reusing the Qwen3-ASR encoder with an MLP-tower adapter and an 8-layer Qwen2-style decoder (CC-BY-NC, local conversion only)
audio8_tts	TTS, Clone	auto, yue, zh, nl, en, fr, de, it, ja, ko, pl, es	GGUF Q8, Stream	@jasonchen31	Audio8 TTS Preview 0.6B DualAR multilingual TTS and zero-shot voice cloning with a Qwen backbone and neural codec
auk	TTS, audio editing	auto	GGUF F32/F16/Q8	@0xShug0	AuK Base and AuK-Flash instruction-guided speech generation and editing with separate generator, Qwen conditioner, and VAE components
chatterbox_turbo	TTS (testing)	en	GGUF 16/Q8	@pannagaps	Chatterbox Turbo distilled 350M GPT2 T3 backbone + 2-step meanflow S3Gen decoder; built-in voice
confucius4_r2t2	ASR	30 languages	Safetensors, GGUF F16/Q8/Q4, Stream	@davidxifeng

Prebuilt Binaries

Official packages for every release are attached to the Releases page:

Platform	Backends
Windows x64	CPU, Vulkan, CUDA (12.4 / 13.3)
Ubuntu x64	CPU, Vulkan
macOS (arm64 / x64)	Metal

The Windows CUDA packages ship the CUDA runtime in a separate cudart archive. Extract it next to the binaries so the CUDA backend can load ggml-cuda.dll and the CUDA runtime DLLs (cudart, cuBLAS, cuFFT).

• Windows (HIP/ROCm, AMD GPUs): community-maintained packages with the ROCm runtime bundled, so no HIP SDK installation is required. Published from @IIIIIllllIIIIIlllll's fork Releases in two tracks: ROCm 6.4 (full coverage incl. RX 7600 / gfx1102) and ROCm 7.1 (recommended for RDNA4). Version numbers follow the upstream releases; see docs/build/windows-hip-distribution.md for details.

Homebrew Install

On macOS, audio.cpp can be installed from the Homebrew tap:

brew tap 0xShug0/audio-cpp
brew trust 0xShug0/audio-cpp
brew install audio-cpp

WebUI

audiocpp_server includes an embedded SvelteKit/TypeScript WebUI for running local TTS, cloning, ASR, generation, conversion, separation, VAD, diarization, and alignment workflows. The production UI is compiled into the server binary, so using it requires neither Python nor separate frontend files:

audiocpp_server --ui --ui-management --backend cuda

Open http://127.0.0.1:8080. --ui-management enables catalog browsing, downloads, temporary uploads, and dynamic model switching, so this is the easiest way to try audio.cpp without writing a server config first. For a locked-down config-driven instance, start with --ui and a server config instead; in that mode the UI only offers models declared by the config unless management is enabled.

For Docker UI downloads, mount a writable models directory at /app/models and use --ui-management. See docker.md.

The native UI also exposes background model download/preparation, long-text split-and-merge synthesis, a browser-local saved voice library, microphone recording, and near-live ASR input. Some model preparation jobs invoke the repository's Python model manager because those packages require Hugging Face download or checkpoint conversion; model inference and the embedded UI remain Python-free. See webui/README.md for launch commands, model notes, and frontend development instructions.

Docker

Docker CUDA and CPU images are available for both CLI and server use. See docker.md for available images, build commands and working Docker examples.

Model Manager and GGUF Downloads

Use tools/model_manager_v2.py for normal model downloads. It reads model_specs/*.json and installs the default package for each family, preferring ready-to-use GGUF packages when they are available.

Native builds configured with -DAUDIOCPP_BUILD_NATIVE_MODEL_MANAGER=ON also provide audiocpp_model_manager, a standalone headless frontend over the same reusable C++ package-management library used by the server and embedded WebUI. It supports list, info, install, clean, and remove without starting audiocpp_server; this is the preferred native path for CLI, Docker, remote provisioning, and other scripted environments. The Python v2 manager remains available as an alternative during migration. The native path uses bundled TLS by default and does not require libcurl.

audiocpp_model_manager list
audiocpp_model_manager install qwen3_asr_0_6b_q8_0 --models-dir models

The old safetensors/converter catalog has been renamed to tools/model_manager_deprecated.py. Use it only for legacy model layouts that have not moved to spec-backed GGUF packages yet.

GGUF downloads:

• Released model packages: audio-cpp/audio.cpp-gguf
• Community model package: mirek190/audio.cpp

See the Model Manager guide for model-manager usage and package notes.

Build yourself

OS	Requirements
Linux	GCC 13 or newer, CMake, plus the backend toolchain for the build you want: NVIDIA CUDA Toolkit for CUDA, Vulkan SDK for Vulkan, ROCm for HIP
Windows	Visual Studio Build Tools 2022 or newer with C++ desktop workload, MSVC x64 compiler, Windows SDK, CMake, Ninja, MSVC OpenMP components; official NVIDIA CUDA Toolkit for CUDA builds, Vulkan SDK for Vulkan builds, AMD HIP SDK for HIP builds
macOS	Xcode or Xcode Command Line Tools, plus CMake. Metal builds also require the Metal compiler available through xcrun

Composite builds let you compile only the model families you need. full is the default and is what release/Docker builds should use. custom registers only the requested loaders while still linking required internal dependencies; core builds the runtime without the optional model-family set.

The helper scripts expose this as --model-set and --models on Linux/macOS, and -ModelSet and -Models on Windows:

scripts/build_linux.sh --backend cuda --model-set custom --models qwen3_tts,pocket_tts,qwen3_asr --target audiocpp_cli

Direct CMake builds use the same underlying variables:

cmake -S . -B build/debug -DCMAKE_BUILD_TYPE=Debug -DAUDIOCPP_MODEL_SET=custom -DAUDIOCPP_MODELS=qwen3_tts,pocket_tts,qwen3_asr
cmake --build build/debug --target audiocpp_cli -j 8

Use the Linux helper script for CPU, CUDA, Vulkan, or HIP builds:

scripts/build_linux.sh --backend cuda --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend vulkan --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend hip --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend cpu --target audiocpp_cli --target audiocpp_server

The script writes to aligned build directories such as build/linux-cuda-release, build/linux-vulkan-release, build/linux-hip-release, and build/linux-cpu-release.

Without --cuda-arch, CUDA builds use the portable arch list (works on many GPUs, slower to build). For a faster build targeting only the local GPU, pass --cuda-arch native (CMake >= 3.24; on older CMake this falls back to the portable list) or an explicit arch list:

scripts/build_linux.sh --backend cuda --cuda-arch native --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend cuda --cuda-arch "86;89" --target audiocpp_cli --target audiocpp_server

Nothing caches compiler output by default. --ccache wires ccache up as the compiler launcher, which leaves a cold build about as slow as it already is and makes a rebuild roughly 14x faster. A CMAKE_<LANG>_COMPILER_LAUNCHER already set in the environment is left alone.

scripts/build_linux.sh --backend cuda --cuda-arch native --ccache --target audiocpp_cli

Composite examples:

scripts/build_linux.sh --backend cuda --model-set full --target audiocpp_cli
scripts/build_linux.sh --backend cuda --model-set custom --models qwen3_tts,pocket_tts,qwen3_asr --target audiocpp_cli
scripts/build_linux.sh --backend cpu --model-set core --target audiocpp_cli

For portable CPU kernels on machines where native ISA flags are not suitable:

scripts/build_linux.sh --backend cuda --native-cpu OFF --target audiocpp_cli --target audiocpp_server

For deployment builds with compiled package specs:

scripts/build_linux.sh --backend cuda --deployment-build --target audiocpp_cli --target audiocpp_server

For native WebUI model downloads, enable the native model manager:

scripts/build_linux.sh --backend cuda --native-model-manager --target audiocpp_server

For direct CMake commands, see docs/build/linux.md. For the opt-in Rockchip NPU (RKNN) decoder path, see docs/build/rockchip-npu.md.

Use the Windows PowerShell build script:

powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\scripts\build_windows.ps1

Common presets:

.\scripts\build_windows.ps1 -Preset windows-cuda-release -Target audiocpp_cli
.\scripts\build_windows.ps1 -Preset windows-vulkan-release -Target audiocpp_cli
.\scripts\build_windows.ps1 -Preset windows-cpu-release -Target audiocpp_cli
.\scripts\build_windows.ps1 -Target audiocpp_server -Jobs 16
.\scripts\build_windows.ps1 -Preset windows-cuda-release -ModelSet custom -Models "qwen3_tts,pocket_tts,qwen3_asr" -Target audiocpp_cli

From cmd.exe, use the wrapper:

scripts\build_windows.cmd

For deployment builds with compiled package specs:

.\scripts\build_windows.ps1 -DeploymentBuild -Target audiocpp_cli

For native WebUI model downloads, enable the native model manager:

.\scripts\build_windows.ps1 -NativeModelManager -Target audiocpp_server

For requirements, CPU profiles, CUDA packaging, and release zips, see docs/build/windows.md.

CPU-only Build

Apple builds enable Metal by default. To build a CPU-only binary, disable Metal explicitly:

cmake -S . -B build/macos-cpu-release \
  -DCMAKE_BUILD_TYPE=Release \
  -DENGINE_ENABLE_CUDA=OFF \
  -DENGINE_ENABLE_VULKAN=OFF \
  -DENGINE_ENABLE_METAL=OFF \
  -DENGINE_ENABLE_OPENMP=OFF \
  -DGGML_OPENMP=OFF
cmake --build build/macos-cpu-release \
  --parallel "$(sysctl -n hw.logicalcpu)" \
  --target audiocpp_cli audiocpp_server audiocpp_gguf

Confirm that the resulting CLI sees the host CPU backend:

build/macos-cpu-release/bin/audiocpp_cli --list-devices

Metal Build

On macOS, use the Metal helper script to build against ggml's Metal backend:

scripts/build_metal.sh --target audiocpp_cli

The script configures build/macos-metal-release by default, enables ENGINE_ENABLE_METAL=ON, disables CUDA and Vulkan, embeds the Metal shader library, and builds static libraries plus the requested target.

Useful variants:

scripts/build_metal.sh --target audiocpp_server
scripts/build_metal.sh --build-type Release --archs arm64 --target audiocpp_cli
scripts/build_metal.sh --model-set custom --models qwen3_tts,pocket_tts --target audiocpp_cli
scripts/build_metal.sh --with-tests --target audio_dsp_test
scripts/build_metal.sh --openmp auto --target audiocpp_cli
scripts/build_metal.sh --native-cpu OFF --target audiocpp_cli
scripts/build_metal.sh --deployment-build --target audiocpp_cli

The built CLI is written to:

build/macos-metal-release/bin/audiocpp_cli

On Linux and Windows, HIP builds compile ggml's CUDA backend sources as HIP code for AMD GPUs. ENGINE_ENABLE_HIP and ENGINE_ENABLE_CUDA are mutually exclusive — configure with exactly one of them.

Linux (the helper script auto-detects ROCm via ROCM_PATH/HIP_PATH/hipconfig and local GPU targets via amdgpu-arch, falling back to rocminfo; pass --gpu-targets to build for other architectures, or when no AMD GPU is visible, e.g. in a VM or container):

scripts/build_linux.sh --backend hip --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend hip --gpu-targets "gfx1100;gfx1103" --target audiocpp_cli

Direct CMake:

cmake -S . -B build_hip \
  -DENGINE_ENABLE_HIP=ON \
  -DGPU_TARGETS=gfx1151 \
  -DCMAKE_C_COMPILER="$(hipconfig -l)/clang" \
  -DCMAKE_CXX_COMPILER="$(hipconfig -l)/clang++" \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build_hip -j$(nproc)

Windows (the helper script auto-detects ROCm, GPU targets, cmake, and ninja):

powershell -ExecutionPolicy Bypass -File scripts\build_windows_hip.ps1

Run with --backend hip (rocm is accepted as an alias). For GPU target selection, hipBLASLt GEMM notes, iGPU tuning, and known limitations, see docs/build/HIP.md.

For Nix and NixOS builds, see the build guide.

Optional server adapters are built from the external/audio.cpp-server-frontends submodule. They are separate from the embedded WebUI. From the repository root, fetch the package (using HTTPS here so no GitHub SSH key is required):

git -c 'url.https://github.com/[email protected]:' submodule update --init external/audio.cpp-server-frontends

For example, build a server with audio decoding and MP3 encoding adapters:

cmake -S . -B build/debug \
  -DCMAKE_BUILD_TYPE=Debug \
  -DAUDIOCPP_BUILD_SERVER_FRONTENDS=ON \
  -DAUDIOCPP_SERVER_FRONTENDS_DIR=external/audio.cpp-server-frontends \
  -DAUDIOCPP_SERVER_FRONTEND_MODULES="audio_decode;mp3_encode"
cmake --build build/debug --target audiocpp_server --parallel 8

The example uses the default backend settings of a fresh build directory; add the backend flags from the platform build instructions when needed. Module-specific dependencies and available adapters are documented in the frontend package. See the server guide for configuration and usage.

Option	Meaning	Default
ENGINE_ENABLE_CUDA	Enable the ggml CUDA backend. Required for --backend cuda.	OFF
ENGINE_ENABLE_HIP	Enable the ggml HIP backend (AMD GPUs). Required for --backend hip; mutually exclusive with ENGINE_ENABLE_CUDA.	OFF
ENGINE_ENABLE_VULKAN	Enable the ggml Vulkan backend. Required for --backend vulkan.	OFF
ENGINE_ENABLE_METAL	Enable the ggml Metal backend. Required for --backend metal.	OFF on most platforms, ON on Apple
ENGINE_ENABLE_LLAMAFILE	Enable llamafile SGEMM support in ggml CPU builds.	ON
ENGINE_ENABLE_CUDA_GRAPHS	Enable ggml CUDA graphs support when CUDA is enabled.	ON
ENGINE_ENABLE_NATIVE_CPU	Build ggml CPU kernels with native host ISA flags such as -march=native. Disable this for portable CPU kernels or toolchains that reject generated CPU instructions.	ON
ENGINE_ENABLE_OPENMP	Enable OpenMP for host-side parallel work.	ON
ENGINE_BUILD_EXAMPLES	Build example binaries.	OFF
AUDIOCPP_BUILD_SERVER_FRONTENDS	Enable optional in-process server frontend adapters, separate from the embedded WebUI.	OFF
AUDIOCPP_SERVER_FRONTEND_MODULES	Semicolon-separated adapter modules to build. Requires AUDIOCPP_BUILD_SERVER_FRONTENDS=ON and an external frontend package.	empty
AUDIOCPP_SERVER_FRONTENDS_DIR	External frontend package directory containing server_frontends.cmake, which registers the requested modules.	empty
ENGINE_BUILD_TESTS	Build framework unit tests.

Usage

[!IMPORTANT] Please read your model's documentation: open the model-docs index, choose your task category, then find your model. Also read the dedicated CLI usage guide and server guide. The examples below are quick starts; model requirements, supported options, and setup details vary.

CLI

The main CLI binary is:

build/bin/audiocpp_cli

High-level command shape:

audiocpp_cli --task <task> --model <path> --family <family> [--backend <backend>] [--mode <mode>] [options]

Always specify --family for model commands, including model-specific help. Treat it as required; omission is supported only for backward compatibility. Automatic loader discovery without it can substantially slow startup and --model ... --help.

Core selectors:

• --task vad|asr|diar|sep|gen|tts|clon|vc|s2s|align|vdes|spk|svc|midi|wake|cls
• --model <path>
• --family <name> explicitly selects the model family
• --backend cpu|cuda|vulkan|metal|best
• --mode offline|streaming; streaming is available for models whose docs list streaming support

Common interface options:

• --load-option key=value passes model-load options, such as PocketTTS language selection
• --session-option key=value passes session/runtime options, such as backend-specific weight controls
• --request-option key=value passes per-request model options
• --config <id> selects a discovered config asset
• --weight <id> selects a discovered weight asset
• --device <n> selects the backend device
• --threads <n> sets backend and OpenMP worker threads

Examples:

Text-to-speech:

build/bin/audiocpp_cli \
  --task tts \
  --family pocket_tts \
  --model /path/to/model \
  --backend cuda \
  --text "audio.cpp is running PocketTTS locally." \
  --voice-ref assets/resources/sample.wav \
  --out build/out/pocket_tts.wav

PocketTTS with another language and a built-in voice:

build/bin/audiocpp_cli \
  --task tts \
  --family pocket_tts \
  --model /path/to/models/pocket-tts \
  --backend cuda \
  --load-option language=spanish \
  --text "Hola, esta es una prueba corta de Pocket TTS." \
  --voice-id alba \
  --out build/out/pocket_tts_spanish.wav

ASR:

build/bin/audiocpp_cli \
  --task asr \
  --family qwen3_asr \
  --model /path/to/model \
  --backend cuda \
  --audio assets/resources/sample_16k.wav

Voice conversion:

build/bin/audiocpp_cli \
  --task vc \
  --family seed_vc \
  --model /path/to/model \
  --backend cuda \
  --audio assets/resources/a.wav \
  --voice-ref assets/resources/b.wav \
  --out build/out/seed_vc.wav

Useful CLI features:

• --help with --task shows task-oriented help
• --help with --model <path> and --family <family> shows model-owned request, session, and load options
• --inspect prints discovered configs, weights, and capabilities
• --list-loaders prints registered model families (--json for the machine-readable contract)
• --list-devices prints backend devices and the --backend/--device values that select them (--json adds PCI ids and free/total memory)
• python tools/model_manager_v2.py list --json prints installable packages from model_specs/*.json
• --batch-text-file <txt> runs one offline request per non-empty line
• --batch-text-dir <dir> runs one offline request per .txt, .md, or .json file, normalizing each file as one paragraph
• --batch-audio-dir <dir> runs one offline request per .wav
• --audio-chunk-mode auto lets ASR/alignment models choose their safe long-audio policy; expert users can override with fixed, vad, or none where supported
• --request-sequence <json> runs a multi-request offline session
• --batch-merge-audio none|concat controls batch audio merge behavior
• --batch-manifest-out <json> writes a batch output manifest
• --metrics prints compact offline wall time, audio duration, RTF, realtime speed, sample rate, and channel metrics
• Use --request-sequence <json> --metrics for per-request metrics from one long-lived offline session
• --pipeline <json> runs a workflow instead of a raw task
• --list-pipelines prints registered workflows
• --workflow-input key=value overrides pipeline inputs
• --recipe <name> runs a built-in multi-model composition without a workflow file; see Built-in recipes
• streams framework logs to stdout

Server

The server binary is:

build/bin/audiocpp_server

Build:

cmake --build build -j$(nproc) --target audiocpp_server

Create a config file with your own model paths:

cat > server.json <<'JSON'
{
  "host": "127.0.0.1",
  "port": 8080,
  "backend": "cuda",
  "device": 0,
  "threads": 1,
  "lazy_load": true,
  "models": [
    {
      "id": "pocket-tts",
      "family": "pocket_tts",
      "path": "/path/to/models/pocket-tts",
      "task": "tts",
      "mode": "offline",
      "load_options": {
        "language": "english"
      },
      "session_options": {
        "language": "english"
      }
    },
    {
      "id": "qwen3-asr",
      "family": "qwen3_asr",
      "path": "/path/to/models/Qwen3-ASR-0.6B",
      "task": "asr",
      "mode": "offline"
    }
  ]
}
JSON

Set "lazy_load": true to register configured model ids at startup while loading each model only on first use. Use per-model "lazy": true or "lazy": false to override that default.

Set top-level "backend" to "cuda", "cpu", "vulkan", "metal", or "hip". CUDA is the optimized path for audio.cpp; CPU, Vulkan, Metal, and HIP are intended for portability and testing when the binary is built with that backend, but performance and model coverage may be lower.

[!WARNING] Lazy loading does not unload models after a request. Once a model is first used, the server keeps that model and session in memory for reuse until the server exits, unless max_loaded_models limits residency.

Set top-level "max_loaded_models" (or start with --max-loaded-models <n>) to bound how many models stay resident in memory at once: loading one more past the limit first unloads the least recently used idle model, and 1 enforces a single loaded model at a time. The default 0 keeps every used model in memory. See app/server/README.md for details.

Start:

build/bin/audiocpp_server --config server.json

The server exposes:

• GET /health
• GET /v1/models
• POST /v1/audio/speech
• POST /v1/audio/transcriptions
• POST /v1/audio/transcriptions/details
• POST /v1/audio/alignments
• POST /v1/tasks/run
• POST /v1/tasks/batch

More server examples are in app/server/README.md.

Pipelines

Pipelines are an experimental JSON workflow feature for chaining multiple model and audio-processing steps behind one CLI command. A pipeline can define default inputs, let users override them with --workflow-input key=value, split long media into model-sized chunks, merge text or audio outputs back together, write intermediate artifacts under --out-dir, and copy the declared final_audio to --out.

This is the higher-level layer for production-style audio jobs: redubbing, batch cleanup, long-form narration, voice conversion, source-separation workflows, transcription-plus-alignment, and future workflows that combine translation, diarization, denoise, enhancement, or review steps as those model surfaces are wired into the framework.

The included same-language speech redub pipeline transcribes long speech in chunks with Qwen3 ASR, merges the transcript, then regenerates the speech in a target reference voice with Qwen3 TTS. The default test input assets/resources/speech.wav is about 418 seconds long and was generated from an 8,091-character speech text, so it exercises long-audio split and merge behavior rather than a short one-shot request:

build/bin/audiocpp_cli \
  --pipeline assets/pipeline/speech_redub.json \
  --backend cuda \
  --out-dir build/out/speech_redub_pipeline \
  --out build/out/speech_redub_pipeline.wav

Override the source speech or target voice without editing the JSON:

build/bin/audiocpp_cli \
  --pipeline assets/pipeline/speech_redub.json \
  --backend cuda \
  --workflow-input source_audio=/path/to/speech.wav \
  --workflow-input target_voice=/path/to/voice.wav \
  --workflow-input language=English \
  --out-dir build/out/speech_redub_pipeline \
  --out build/out/speech_redub_pipeline.wav

Tests

The repository includes both framework-level parity validation and app-level end-to-end path checks. At a high level, the flow is:

The main harness under tests/ is tests/warmbench.py. It is used for long-lived multi-request validation, parity checks against Python references, and performance-oriented session reuse scenarios. The tests/ tree also contains model-specific C++ and Python warmbench entrypoints that warmbench.py coordinates.

The main app-facing test tooling under tools/ is tools/audiocpp_cli/run_audiocpp_cli_path_tests.py. It drives audiocpp_cli through cataloged offline and streaming cases, verifies expected outputs such as audio or JSON artifacts, and is useful for checking real user-facing request paths rather than just lower-level model components. Streaming coverage is model-specific and applies to models documented with streaming support.

The Python-reference side of these tests usually requires more time-consuming setup than the C++ path because different models rely on different Python reference repos and dependency stacks. In practice, the framework-side tooling is fast to iterate on once models are installed, while Python parity runs often need extra environment preparation before they are ready.

Projects

Last update: 2026-09-23

Have a project using audio.cpp? Submit a PR or let me know, and I’ll be happy to add it here.

• AIRI Audio Server: A lightweight, zero-Python Node.js microservice wrapper providing OpenAI-compatible /v1/audio/speech (with real-time SSE streaming), /v1/audio/transcriptions (Citrinet/Whisper), dynamic voice catalog discovery, and serialized GPU queueing for audio.cpp.
• Project AIRI: An open-source, next-generation AI companion and desktop frontend featuring interactive Voice Studio curation, zero-shot voice cloning dropzones, and real-time audio pipeline integration powered by airi-audio-server and audio.cpp.
• TranscrIA is a self-hosted meeting transcription platform with diarization and local LLM correction. audio.cpp is integrated as a first-class STT engine in the product.
• Pocket TTS Browser Engine uses audio.cpp to bring fully local PocketTTS voices into Chrome and Edge through the browser TTS API.
• GuideAnts uses audio.cpp as the default local AI stack path for basic ASR and TTS, with planned reusable skills for audio.cpp scenarios and model configurations.
• audio.cpp-webui provides a full-task Python/Gradio WebUI for audio.cpp, focused on browser-based model downloads and common TTS, ASR, voice conversion, diarization, music, and audio workflows.
• audio.cpp-hub is a Web GUI for audio.cpp that packages model-oriented workflows around the native runtime.
• Delusion provide Pythonic, strongly typed wrapper classes around audio.cpp usage, including model download/cache helpers and typed request surfaces.
• AudioCppTray is a Windows tray management tool for audiocpp_server.exe, with start/stop/restart controls, notifications, log viewing, log rotation, and server configuration shortcuts.

Performance Metrics

[!WARNING] These Python-relative numbers were measured for the initial release. Several model paths have improved substantially since then, so the figures below should be read as the original release baseline rather than the latest peak performance.

All performance metrics in this section were measured on Ubuntu with the CUDA backend on an NVIDIA GeForce RTX 5090. The Python-relative one-shot and long-lived-session comparisons come from direct framework/runtime API benchmark calls, not from audiocpp_cli; CLI path tests are separate and include app-layer request parsing, output writing, and other user-facing overhead.

Absolute RTF depends on the GPU and system setup, but the Python-relative speedups are real because audio.cpp and the matching Python reference paths were measured on the same CUDA setup.

audio.cpp already shows some genuinely exciting wins against the matching Python reference paths, especially on the TTS side, even when using the original model weights without quantization. The headline win is wall time: several TTS paths run 1.8x to up to 10x faster than Python while cutting end-to-end latency by 45%-90%.

• In one-shot runs, several TTS-family models already land far ahead of Python:
  • vevo2: 5.03x faster with 80.11% less wall time
  • pocket tts: 3.68x faster with 72.80% less wall time
  • miotts: 2.73x faster with 63.39% less wall time
  • moss_tts_local: 2.33x faster with 57.07% less wall time
  • qwen3 tts: 1.83x faster with 45.34% less wall time
  • vibevoice: 1.40x faster with 28.75% less wall time
• In long-lived-session runs, where the same loaded session serves multiple requests in sequence, the gains stay strong:
  • pocket tts: 3.22x faster with 68.91% less wall time
  • qwen3 tts: 2.74x faster with 63.47% less wall time
  • moss_tts_local: 2.66x faster with 62.35% less wall time
  • miotts: 2.28x faster with 56.22% less wall time
  • vibevoice: 1.77x faster with 43.55% less wall time
  • vevo2: 1.75x faster with 42.72% less wall time
• In long-form runs on the shared 6,026-character, 1,028-word passage, the strongest Python-relative wins still show up clearly:
  • pocket tts: 3.15x faster with 68.23% less wall time
  • qwen3 tts: 3.06x faster with 67.33% less wall time
  • vibevoice: 2.86x faster with 65.07% less wall time
  • vevo2: 1.77x faster with 43.51% less wall time
  • chatterbox: 1.58x faster with 36.83% less wall time
• These long-lived-session numbers are especially important for real applications, because they reflect the common case where model load, cached state, and reusable runtime setup are amortized across many requests.

The figures report Python wall time / audio.cpp wall time. The 1.0x line means equal wall time; bars above 1.0x mean audio.cpp is faster than Python, and bars below 1.0x mean it is slower.

For TTS-family models, the measured one-shot RTF is:

model	audio len (s)	wall time (s)	RTF	x faster than real time
chatterbox	9.72	2.45	0.252	3.97x
miotts	20.40	3.30	0.162	6.18x
moss_tts_local	9.60	0.97	0.101	9.91x
omnivoice	9.00	1.32	0.146	6.84x
pocket tts	8.08	0.26	0.032	31.09x
qwen3 tts	11.44	4.46	0.390	2.56x
vevo2	8.66	2.47	0.285	3.51x
vibevoice	11.07	5.02	0.454	2.20x
voxcpm2	5.60	3.09	0.551	1.81x

For long-form TTS tests, each run uses the same 6,026-character, 1,028-word input text (vibevoice uses 106,310 chars, 18,052 words, 4 speakers). Rows are CUDA unless marked CPU. The measured RTF is:

model	audio len (s)	wall time (s)	RTF	x faster than real time
chatterbox	391.24	58.57	0.150	6.68x
index tts2	422.12	139.95	0.332	3.02x
miotts	399.16	66.59	0.167	5.99x
moss_tts_nano	391.20	43.16	0.110	9.06x
moss_tts_local	375.44	73.84	0.197	5.08x
omnivoice	357.00	17.77	0.050	20.09x
pocket tts	353.12	7.30	0.021	48.40x
qwen3 tts	327.60	72.65	0.222	4.51x
supertonic	379.32	2.02	0.005	187.62x
supertonic (CPU)	379.40	61.40	0.162	6.18x
vevo2	457.68	52.47	0.115	8.72x
voxcpm2	315.84	72.70	0.230	4.34x
vibevoice	5615.73	1376.84	0.245	4.08x

Runtime Memory Options

Some models expose memory-saver session options such as ace_step.mem_saver=true, dramabox.mem_saver=true, heartmula.mem_saver=true, stable_audio.mem_saver=true, omnivoice.mem_saver=true, voxcpm2.mem_saver=true, and echo_tts.mem_saver=true. These options keep the default output path unchanged while reducing graph workspace VRAM or releasing staged graph/cache state after request phases; later requests may rebuild released graphs.

Precision/Quantization Support

Many model sessions expose quantization through --session-option <family>.weight_type=<mode>, and some families also expose more specific knobs such as ...conv_weight_type, ...talker_weight_type, or ...speech_decoder_weight_type. The exact supported modes are model-specific rather than global.

audio.cpp also supports standalone GGUF packages.

In practice, lower precision and quantized modes should be treated as model- and route-specific optimizations rather than universally safe defaults.

• Safety. Quantization may not be safe on every path even when a model parser accepts the option. For example, in our ACE-Step 1.5 checks, lower-precision runs could fail at runtime with ACE-Step planner masked decode found no valid token while higher-precision settings completed normally.

• Quality Drop. Output quality can drop a lot. In our VeVo2 checks, non-fp32 outputs showed noticeably weaker similarity to the fp32 reference under the repo's existing waveform and log-mel comparison metrics, and even output length could shift.

• Performance Gain. The performance gain may be minor relative to that quality risk. For example, q8_0 was faster than the default setting by only around 3.8% on Qwen3-TTS and around 3.6% on VeVo2. Other models may benefit more, but the tradeoff should be validated per model and per route rather than assumed.

• Memory Benefit. Lower precision and quantized weights can still be useful for reducing weight memory footprint and making larger models easier to fit within device limits. For example, in our Qwen3-TTS checks, switching from the default setting to q8_0 reduced peak RAM by about 3.7% and peak VRAM by about 25.0%. That benefit is real, but it should be evaluated together with runtime stability, output quality, and end-to-end speed rather than assumed from precision alone.

Notes

• The repo supports multiple backends, but backend and model coverage are model-dependent.
• GGUF is a container, not a universal architecture adapter. Existing llama.cpp or whisper.cpp GGUF files are not automatically compatible unless their tensor names and model metadata are mapped to the audio.cpp family implementation.
• Build_xcframework.sh is outdated; Metal and Apple XCFramework packaging still need to be retested after the framework refactor.