oMLX

---

Every LLM server I tried made me choose between convenience and control. I wanted to pin everyday models in memory, auto-swap heavier ones on demand, set context limits - and manage it all from a menu bar.

oMLX persists KV cache across a hot in-memory tier and cold SSD tier - even when context changes mid-conversation, all past context stays cached and reusable across requests, making local LLMs practical for real coding work with tools like Claude Code. That's why I built it.

Install

macOS App

Download the .dmg from Releases, drag to Applications, done. The app includes in-app auto-update, so future upgrades are just one click. The macOS app also installs a lightweight ~/.omlx/bin/omlx CLI shim so terminal commands and Apple Shortcuts can control the app-managed server.

Homebrew

brew tap jundot/omlx https://github.com/jundot/omlx
brew install jundot/omlx/omlx --with-custom-kernel   # With native custom kernels (needs full Xcode)
# No full Xcode? brew install jundot/omlx/omlx installs without the kernels

# Upgrade to the latest version
brew update && brew upgrade omlx

# Run as a background service (auto-restarts on crash)
omlx start

# Optional: MCP (Model Context Protocol) support, then restart to load it
/opt/homebrew/opt/omlx/libexec/bin/pip install mcp
omlx restart

From Source

git clone https://github.com/jundot/omlx.git
cd omlx
make install                # Editable install with the web UI and native custom kernels
# No full Xcode? make install-no-kernels installs without the kernels
make mcp                    # Optional: MCP support, restart `omlx serve` to load it

Requires macOS 15.0+ (Sequoia), Python 3.11–3.13, and Apple Silicon (M1/M2/M3/M4/M5).

Note on native custom kernels: GLM-5.2, MiniMax M3, and Qwen3.5 need them. Without them those families silently fall back to much slower generic paths -- for GLM-5.2 the fused DSA prefill is roughly 30x faster with the kernels (measured 845 vs ~29 tok/s on an M3 Ultra), and the fallback also uses more memory (#2137). make install builds them and checks that each one loads. That needs full Xcode with the Metal toolchain (xcodebuild -downloadComponent MetalToolchain); Command Line Tools alone do not provide it (xcrun: error: unable to find utility "metal"). Without Xcode, use the official DMG, which ships the kernels precompiled, or make install-no-kernels. A plain pip install -e . also skips them. make kernels rebuilds only the kernels from scratch. Homebrew builds them with --with-custom-kernel, which also needs full Xcode. To verify any install:

python -c "from omlx.custom_kernels import native_kernel_status; print(native_kernel_status())"

Quickstart

macOS App

Launch oMLX from your Applications folder. The Welcome screen guides you through three steps - model directory, server start, and first model download. That's it. To connect OpenClaw, OpenCode, Codex, Hermes Agent, Copilot, or DeepSeek Harness, see Integrations.

CLI

# Managed background server (macOS app or Homebrew install)
omlx start
omlx stop
omlx restart

# Foreground server attached to this terminal
omlx serve --model-dir ~/models

The server discovers LLMs, VLMs, embedding models, and rerankers from subdirectories automatically. Any OpenAI-compatible client can connect to http://localhost:8000/v1. A built-in chat UI is also available at http://localhost:8000/admin/chat.

Homebrew Service

If you installed via Homebrew, you can run oMLX as a managed background service:

omlx start                    # Start via brew services
omlx stop                     # Stop
omlx restart                  # Restart

brew services start omlx    # Start (auto-restarts on crash)
brew services stop omlx     # Stop
brew services restart omlx  # Restart
brew services info omlx     # Check status

The service runs omlx serve with zero-config defaults (~/.omlx/models, port 8000). omlx start, omlx stop, and omlx restart are the portable lifecycle commands; Homebrew installs delegate them to brew services. To customize, either set environment variables (OMLX_MODEL_DIR, OMLX_PORT, etc.) or run omlx serve --model-dir /your/path once to persist settings to ~/.omlx/settings.json.

Logs are written to two locations:

• Service log: $(brew --prefix)/var/log/omlx.log (stdout/stderr)
• Server log: ~/.omlx/logs/server.log (structured application log)

Features

Supports text LLMs, vision-language models (VLM), OCR models, embeddings, and rerankers on Apple Silicon.

Admin Dashboard

Web UI at /admin for real-time monitoring, model management, chat, benchmark, and per-model settings. Supports English, Korean, Japanese, Chinese, French, Russian, Spanish, and Brazilian Portuguese. All CDN dependencies are vendored for fully offline operation.

Experimental Multi-Mac Inference

Source builds can split one downloaded language model across unequal-memory Macs using MLX pipeline ranks over Ring or Thunderbolt RDMA/JACCL. The Cluster dashboard handles read-only peer discovery, strict SSH/runtime verification, byte-aware unequal shard planning, measured compute/link rebalancing, headroom-aware execution tuning, activation, and a live shard/performance map on both Macs. Interactive, balanced, and throughput profiles expose coalesced batching, prompt-cache affinity, rotating-KV limits, Ring connection tuning, and a capability-gated experimental token-only output path. See Distributed inference across Macs for setup, security boundaries, current limitations, and the physical-hardware validation checklist.

This is the beginning of the README. Read the rest at the original. Read the full README ↗