News
- [2026/10] 🗺️ A unified runtime for interactive intelligence: native full-duplex models, turn-based multimodal models, and cascaded systems, coordinated through composable graphs and persistent sessions. [Blog]
- [2026/10] 🚀 SGLang-Omni v0.2.0 is on PyPI. Install with
uv pip install --prerelease=allow "sglang-omni==0.2.0". [Installation]
- [2026/10] 🐟 Day-0 support for Fish Audio Drama 3: multi-speaker text-to-speech with voice cloning and natural-language control, streamed on
/v1/audio/speech. [Blog] [Cookbook] [Technical Report]
- [2026/10] 🎙️ Introducing Voxt, an open-source voice input tool for Mac, now integrating SGLang-Omni for faster local speech recognition on Apple Silicon. Combine ASR with LLM-based text refinement and personal system prompts for accurate technical terminology and personalized voice input, with fully local or remote inference options. [Download][Demo]
- [2026/09] 🐧 Day-0 support for AuK and AuK-Flash: text + voice instructions → 24 kHz speech on
/v1/audio/speech, audio + editing instructions → edited speech on /generate. [Cookbook]
- [2026/08] 🎵 Day-0 support for MiniMax Music 3: lyrics + caption → 32 kHz stereo song on
/v1/audio/speech. [Cookbook]
- [2026/08] 🚀 TTS architecture refactor: shared pipeline state, engine construction, reference encoding, capability metadata, and vocoder scheduling. [Roadmap] [Blog]
- [2026/06] 🔥 MOSS-TTS Local Transformer v1.5 on SGLang-Omni with native-streaming 48 kHz speech. [Blog] [Cookbook]
- [2026/06] 🔥 Higgs Audio v3 TTS for real-time, controllable speech. [Blog] [Cookbook]
About
SGLang-Omni is a multi-stage serving runtime for omni, speech, and TTS models. Its design target is multi-stage decoding: generation split across heterogeneous stages with different compute patterns, dependency structures, and resource needs. SGLang-Omni owns the pipeline topology, stage lifecycle, inter-stage transport, model-family integration layer, and OpenAI-compatible serving surface, while composing with SGLang for high-performance autoregressive scheduling and model execution where applicable.
- Multi-stage runtime: SGLang-Omni models generation as coordinated stages: preprocessing, encoders, autoregressive engines, talkers, decoders, vocoders, and aggregators.
- Stage-specialized scheduling: Each stage runs behind a scheduler matched to its workload, from SGLang-backed autoregressive scheduling to lightweight preprocessing and streaming vocoder loops.
- Transport-aware execution: A control plane coordinates requests while the relay data plane moves tensor payloads across shared-memory, NCCL, NIXL, and Mooncake backends.
- API surface: OpenAI-compatible endpoints expose multimodal chat, speech generation, batch speech, streaming speech, uploaded voices, and transcription.
What SGLang-Omni Serves
- Omni chat and speech: Qwen3-Omni, MiniCPM-o 4.5, Ming-Omni, Ming-3.0-audio-flash — multimodal understanding and text/audio replies.
- Native full-duplex conversation: MiniCPM-o 4.5, PersonaPlex, NemotronLabs VoiceChat — live listening and speaking over
/v1/realtime, including camera input with MiniCPM-o. Full-duplex guide.
- Cascaded voice assistants: Compose ASR → LLM → TTS services into streaming, turn-based conversations with voice activity detection, conversation history, turn taking, and interruption. Streaming cascade guide.
- Image and video generation: LLaDA2.0-Uni for text/image understanding, image generation, and editing; Cosmos3 Nano for interleaved understanding and generation, with native image, video, and action routes.
- Music generation: MiniMax Music 3 — lyrics + caption → 32 kHz stereo song.
Hardware Support
| Backend | Status | Notes |
|---|
| NVIDIA CUDA | Supported | Default backend with full model coverage. |
| Apple Silicon | Experimental | Qwen3-ASR runs through Torch MPS on macOS arm64. Install with install.sh and follow the Qwen3-ASR guide. |
| Intel GPU (XPU) | Experimental | Intel Arc GPUs via PyTorch XPU. Qwen3-ASR, Whisper ASR, Fun-ASR-Nano, Nemotron 3.5 ASR, MOSS-Transcribe-Diarize, Qwen3-TTS, Fun-CosyVoice3, ZONOS2, Audar-TTS, Qwen3-Omni, MiniMax Music 3, MiniCPM-o, Ming-Omni, Ming-Omni-TTS, Nemotron VoiceChat, PersonaPlex, AuK/AuK-Flash and dots.tts serve end-to-end. The Intel XPU guide has the install steps, each model's launch command and how many cards it needs; the backend is auto-detected. |
Additional model guides, including experimental and research-oriented paths, are available in the Cookbook.
Quick Start
Community & Support
SGLang-Omni welcomes contributors working on inference systems, kernels, scheduling, inter-stage communication, model runners and cache efficiency, model integration, benchmarking, production deployment. Join the SGLang Slack or read the developer reference.
Organizations interested in supporting SGLang-Omni, TTS, or omni model serving can contact Chenyang Zhao at [email protected].
Citation
If you use SGLang-Omni for your research, please cite our technical blogs on the unified interactive runtime and the Stage--Process--GPU topology:
@misc{sglangomniinteractive2026,
title={Unified Runtime for Full-Duplex Interactive Intelligence with {SGLang Omni}},
author={{SGLang-Omni Team}},
howpublished={\url{https://www.sglang.io/blog/omni-interactive-runtime}},
year={2026},
month={oct}
}
@misc{sglangomni2026,
title={Fish Audio Drama 3 Meets {SGLang-Omni}: Day-0 Support and the Stage--Process--GPU Topology},
author={{SGLang-Omni Team}},
howpublished={\url{https://www.sglang.io/blog/spg-topology}},
year={2026},
month={oct}
}
Acknowledgments
SGLang-Omni builds on the SGLang ecosystem and on open model work from the TTS, speech, and omni-model communities. We thank the model teams, systems contributors, and partner organizations helping make open multimodal serving faster, more reliable, and easier to extend.
