FluidAudio - Transcription, Text-to-speech, VAD, Speaker diarization with CoreML Models
FluidAudio is a Swift SDK for fully local, low-latency audio AI on Apple devices, with inference offloaded to the Apple Neural Engine (ANE), resulting in less memory and generally faster inference.
The SDK includes state-of-the-art speaker diarization, transcription, and voice activity detection via open-source models (MIT/Apache 2.0) that can be integrated with just a few lines of code. Models are optimized for background processing, ambient computing and always on workloads by running inference on the ANE, minimizing CPU usage and avoiding GPU/MPS entirely.
For custom use cases, feedback, additional model support, or platform requests, join our Discord. We're also bringing visual, language, and TTS models to device and will share updates there.
Below are some featured local AI apps using Fluid Audio models on macOS and iOS:
Want to convert your own model? Check möbius
Highlights
- Automatic Speech Recognition (ASR): Parakeet TDT v3 (0.6b) and other TDT/CTC models for batch transcription supporting 25 European languages and Japanese, plus SenseVoice and Paraformer for Mandarin Chinese, Nemotron 3.5 Transcribe Lite for fast meeting transcripts with speakers, and LFM3-Audio-2B for ASR, live captions and TTS from one audio LM; Parakeet EOU (120m) for streaming ASR with end-of-utterance detection (English only). See all ASR models.
- Inverse Text Normalization (ITN): Post-process ASR output to convert spoken-form to written-form ("two hundred" → "200"). See text-processing-rs. Optional: ASR-only apps can drop the engine (~8 MB per slice) with
traits: [] (Swift 6.2+), see PostProcessing.md
- Text-to-Speech (TTS): Kokoro (82m) for parallel synthesis with SSML and pronunciation control across 9 languages (EN, ES, FR, HI, IT, JA, PT, ZH); PocketTTS for streaming TTS with voice cloning support (EN, DE, ES, FR, IT, PT — 6L and 24L variants); Chatterbox Multilingual (520M, 18 languages) and Chatterbox Nano (110M, English with
[laugh]/[chuckle] paralinguistic tags) in beta — see Documentation/TTS/Chatterbox.md; Paradee (8M Kokoro distill, English, four voices incl. two British, 11 MB) in beta — see Documentation/TTS/Paradee.md; Gradium's Phonon 31m (Apache-2.0, English, 103 voices) on the Neural Engine at ~20x real time — see Documentation/TTS/PocketTTS.md
- Speaker Diarization (Online + Offline): Speaker separation and identification across audio streams. Streaming pipeline for real-time processing and offline batch pipeline with advanced clustering.
- Speaker Embedding Extraction: Generate speaker embeddings for voice comparison and clustering, you can use this for speaker identification
- Voice Activity Detection (VAD): Voice activity detection with Silero models
- Speech Enhancement (AEC + Noise Suppression): LocalVQE (4.8M) removes loudspeaker echo, noise and reverb from 16 kHz mic audio given a far-end reference; streaming with 16 ms latency (beta)
- Apple Neural Engine: Models run efficiently on Apple's ANE for maximum performance with minimal power consumption
- Open-Source Models: All models are publicly available on HuggingFace — converted and optimized by our team; permissive licenses. See full model catalog.
This is the beginning of the README. Read the rest at the original. Read the full README ↗
