M* Documentation

---

What is M*?

M* (pronounced "M-star") is a serving system for the new generation of composite multimodal models — models built from structurally distinct components (vision encoders, transformer backbones, diffusion and flow heads, audio codecs, action generators, world-model predictors) whose execution path changes with the input and the task.

LLM serving stacks assume inference is a single autoregressive loop. Composite models broke that assumption. M*'s core idea is the Walk Graph: a model is a dataflow graph of its components, and every request is a Walk over that graph. A single runtime serves unified multimodal models, omni models, speech LMs, vision-language-action policies, and world models — at or above the performance of engines specialized for each.

Fast — per-component fast paths, matched to each component's bottleneck:

• Paged attention (FlashInfer) and continuous batching for autoregressive backbones
• CUDA-graph capture for encoders and decode
• Classifier-free-guidance parallelism for diffusion / flow
• Tensor parallelism and Ulysses sequence parallelism, composable as a TP × SP mesh per component
• Sliding-window chunk streaming for audio codecs
• Component-level disaggregation with pluggable tensor transport (shared memory, TCP, RDMA)

Flexible — the abstraction mirrors the model:

• One small Python file per model declares its component graph and its Walks
• A YAML file maps components to GPUs at per-component, per-walk granularity — arbitrary disaggregation, no code changes
• Text, image, audio, video, and robot actions, in and out
• A Python SDK, an OpenAI-compatible API, and a native streaming endpoint

Roadmap. M* is evolving toward many-model, agentic multimodal serving — routing requests across many models and tools within one graph-scheduled runtime.

Quickstart

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install --torch-backend=auto "mstar-ai[all]"   # install M* from PyPI
mstar serve bagel          # one command — launch a server (default: http://localhost:8000)

The PyPI package is mstar-ai (mstar-project and mstar-serve install the same thing); the import package and the CLI are mstar. Working from a checkout instead: uv pip install --torch-backend=auto -e .[all]. Qwen3-TTS also needs uv pip install --no-deps "qwen-tts==0.1.1" (see docs/installation.rst).

To enable flash-attn support (required for Qwen3-Omni, recommended for BAGEL),

# torch 2.10 built for CUDA 13.x (cu13); on a cu12 torch, build from source (docs/installation.rst)
uv pip install \
  "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu13torch2.10cxx11abiTRUE-cp312-cp312-linux_x86_64.whl"

Other models: mstar serve flux2_klein · mstar serve cosmos3 · mstar serve cosmos3_droid · mstar serve qwen3_omni · mstar serve kokoro · mstar serve orpheus · mstar serve pi05 · mstar serve vjepa2

Python SDK — works for every model (text, image, audio, video):

from mstar import MStarClient
client = MStarClient("http://localhost:8000")

client.chat("What is the capital of France?").text          # text
client.generate_image("a cat in a hat")                     # → PNG bytes   (FLUX.2 klein, BAGEL, Cosmos3)
client.edit_image("make it a watercolor", "cat.png")        # → PNG bytes   (FLUX.2 klein, BAGEL)
client.tts("Hello there", voice="tara").to_wav("out.wav")   # → speech      (Orpheus)

for event in client.chat("Tell me a story", stream=True):   # streaming
    print(getattr(event, "text", ""), end="", flush=True)

OpenAI-compatible API — drop-in for bagel, cosmos3, qwen3_omni, orpheus, and kokoro:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")

client.chat.completions.create(model="bagel", messages=[{"role": "user", "content": "hi"}])
client.audio.speech.create(model="orpheus", input="hi", voice="tara")   # text-to-speech
client.images.generate(model="bagel", prompt="a cat")                   # image generation

Runnable scripts and curl examples live in examples/. Power users can launch any deployment with an explicit config: mstar-serve --config configs/<model>.yaml.

Note: The first request(s) on a fresh environment can be slow — often tens of seconds to a few minutes. mstar torch.compiles the model on first use, and that compilation happens lazily on the first request that exercises each path.

Supported models

Model	Family	Input → Output	Endpoints
BAGEL	Unified multimodal	text, image → text, image	/v1/chat/completions, /v1/images/generations
Qwen3-Omni	Omni	text, image, audio, video → text, speech	/v1/chat/completions
Qwen3.5 0.8B–27B / Qwen3.8-27B	Vision-language model	text, image → text (hybrid gated-DeltaNet attention)	/v1/chat/completions, /generate
Nemotron VoiceChat (Beta)	Full-duplex speech LM	speech → text, speech (streamed)	/generate
Orpheus	Speech LM	text → speech	/v1/audio/speech
Fish Audio S2-Pro	Speech LM	text, reference audio → speech (streamed; weights under the Fish Audio Research License, non-commercial use only)	/v1/audio/speech
Qwen3-TTS 0.6B / 1.7B	Speech LM	text (+ voice instruction or reference audio) → speech (streamed; CustomVoice, VoiceDesign and Base variants)	/v1/audio/speech
Chatterbox / Multilingual / Turbo

Every model is reachable through the SDK and the native /generate endpoint; the OpenAI-compatible routes cover the chat, speech, transcription, image, and video models. Registry keys, configs and per-model notes are in docs/models.rst.

How it works

HTTP / SDK  →  API Server  →  Conductor  →  Workers (one per GPU)  →  streaming results
                                  │              │
                          walks the graph,   own subgraphs; route tensors
                          schedules walks    directly to one another

A model declares a computation graph of components and a set of named Walks (e.g. prefill, decode, image_gen). The Conductor turns each request into a walk over that graph and schedules it; Workers each own a subgraph on their GPU and stream tensors directly to one another. Logical graph structure is decoupled from physical placement, so the same model runs single-GPU or fully disaggregated by changing only the YAML node_groups. Four composable primitives — Sequential, Parallel, Loop, and a cross-partition StreamingGraphEdge — express every model family above. See the paper for the full design.

Optional: Rust ZMQ transport

The ZeroMQ control mesh can run over a Rust transport (vendored in rust/) instead of pyzmq — wire-compatible, selectable per process with MSTAR_RUST_ZMQ (default AUTO). Note what AUTO-by-default means: on any machine where the extension is built, the whole mesh switches to the Rust transport with no configuration change — each process logs its choice at startup (control mesh transport: ...), and MSTAR_RUST_ZMQ=0 pins pyzmq. Build it with maturin: uv pip install maturin && maturin develop --release -m rust/Cargo.toml. The Linux wheels of mstar-ai on PyPI carry it, with libzmq 4.3.4 statically linked under LGPL-3.0-or-later and its static-linking exception (PACKAGING.md has where its source is). See the installation docs and environment variables.

Performance

Across every model we benchmark, M* matches or beats the system specialized for that family — unified models (BAGEL), omni and speech models (Qwen3-Omni, Orpheus), and world models (Cosmos3, V-JEPA 2) — by executing only the components each request needs and giving each its own fast path: paged attention and continuous batching for autoregressive backbones, classifier-free-guidance parallelism for diffusion, chunk streaming for audio codecs, and persistent-cache loops for world-model rollouts.

Benchmark numbers shift as systems evolve — ours and everyone else's — so rather than freeze figures here that go stale, we keep the current results and full methodology in the blog post and the paper.

Contributing

Issues and pull requests are welcome. Found a bug, or want a model or feature supported? Open an issue. To add a model yourself, follow the Adding a New Model guide. PRs to main go through review and CI (ruff).

Citation

If you use M* in your research, please cite:

@article{mstar2026,
  title  = {M*: A Modular, Extensible, Serving System for Multimodal Models},
  author = {Jha, Atindra and Sagan, Naomi and Kamahori, Keisuke and Sivgin, Irmak and
            Sanda, Rohan and Gao, Steven and Horowitz, Mark and Zettlemoyer, Luke and
            Hsu, Olivia and Leskovec, Jure and Kasikci, Baris and Wang, Stephanie},
  year   = {2026},
  eprint = {2606.12688},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG}
}

From Stanford University & the University of Washington. Correspondence: [email protected].

Acknowledgments

M* builds on ideas and proven primitives from the open-source community — paged attention and continuous batching (vLLM), FlashInfer kernels, streaming speech serving (VoxServe), and RDMA tensor transport (Mooncake).

License

Apache License 2.0.