omni-api

Every local model your app needs, behind one port.

omni is a local CLI. It runs, supervises and serves the models an app needs (chat, decisions, embeddings, images, video, speech-to-text and text-to-speech) on 127.0.0.1:10000. Apps call a role such as chat or tts, never a model name or an engine port. Wherever an OpenAI format exists, omni uses it, so any OpenAI SDK works once you change its base_url.

The first client is Hyperlight.

Screenshots

A 53 s silent tour: why one app needs many models, how omni sets them up, the real playground, and the install commands. Watch the film or open the product page. Every screenshot below is the real playground, in the dark and the light theme (the toggle sits in the top bar; it follows your system until you pick one).

Dark	Light

The playground at /: every role is a card in the left rail (runtime, model, one status that says what it is doing, and the switch that keeps it loaded), each role page is a workbench with a UI / JSON / cURL view of the same request, and that role's live request log sits under it.

Dark	Light

Decide answers noul, choice and score questions as probability distributions, in one forward pass. Embed indexes files, folders, images and audio, then searches them by meaning. Images need a multimodal embed model (embeddinggemma-2, the default).

Dark	Light

Settings → Models lists every role's models, what is on disk, what is loaded and how to add more. Settings → Server holds the gateway, Ollama and worker controls plus the memory and CPU meters.

It fits a phone as well: the role cards become a strip and the panels stack.

Dark	Light

---

How it works

apps ──► 127.0.0.1:10000  omni gateway (FastAPI + httpx streaming proxy)
           /v1/chat/completions      ──► Ollama          OLLAMA_HOST, else 127.0.0.1:11434
           /v1/systemone             ──► Ollama          same instance, scored in the gateway
           /v1/embeddings            ──► Ollama          same instance, EmbeddingGemma task prompts added in the gateway
           /v1/images/generations    ──► image worker    unix:~/.omni-api/run/image.sock
           /v1/videos                ──► video worker    unix:~/.omni-api/run/video.sock
           /v1/audio/transcriptions  ──► stt worker      unix:~/.omni-api/run/stt.sock
           /v1/audio/speech          ──► tts worker      unix:~/.omni-api/run/tts.sock
           /, /static            ──► built playground (src/omni/static/app)

• Gateway (src/omni/gateway/): the FastAPI app (app.py) builds the shared context and mounts one module per route family (routes/). proxy.py streams each request through to its worker with httpx, filling in the default chat model when the request leaves model out.
• Supervisor (src/omni/supervisor/):
  • pool.py spawns each worker in its own process group and probes /health every 2 s.
  • Restarts failed workers with exponential backoff, one lock per worker so a restart and the watch loop never overlap.
  • process.py records each child's pid and command line, so on down only a pid that is still ours is signalled; leftover workers from an earlier run are cleaned up on start.
• Workers (src/omni/workers/): each role runs in its own uvicorn process, so a crash or memory spike in one model never takes down the others. Each loads its model on first use behind one lock; the gateway unloads it once idle, since it sees every request and pin.
• Chat and embed use the machine's own Ollama address (OLLAMA_HOST, else 127.0.0.1:11434). If nothing answers there, omni spawns ollama serve on that same address; it never moves Ollama to another port.
• Decide (src/omni/workers/decide.py, src/omni/inference/vega.py) runs Vega on ONNX Runtime, in its own worker. A frozen Qwen3.5-0.8B reads the state and one typed question; a small physics engine turns the pooled hidden states into option logits; a per-type temperature and a conformal threshold turn those into calibrated probabilities, a prediction set, and abstain and unbound flags. It is not a generative model. The prompt, pooling and calibration are a port of the browser runtime, checked against its parity set (identical token ids and spans; every argmax kept). An image can stand in for the text state (zero-shot: the engine was trained on text features).
• Image (src/omni/workers/image.py) runs QwenImage21Pipeline from diffusers on MPS in bf16. [image] steps sets the default step count.

---

以上為 README 的開頭部分。其餘內容請閱讀原文。 檢視完整 README ↗