⚡️ SwiftLM
A blazingly fast, native Swift inference server that serves MLX models with a strict OpenAI-compatible API.
No Python runtime, no Global Interpreter Lock (GIL), no unnecessary memory copies. Just bare-metal Apple Silicon performance compiled to a single binary.
🏁 Getting Started
Fastest: Download Pre-built Binary
Download the latest release tarball from the Releases page.
The archive is self-contained — mlx.metallib is bundled alongside the binary.
tar -xzf SwiftLM-<version>-macos-arm64.tar.gz
./SwiftLM --model mlx-community/Qwen2.5-3B-Instruct-4bit --port 5413
Build from Source
The build script handles everything: submodules, cmake, Metal kernel compilation, and the Swift build.
git clone --recursive https://github.com/SharpAI/SwiftLM
cd SwiftLM
./build.sh
This will:
- Initialize git submodules
- Install
cmake via Homebrew (if not already installed)
- Compile
mlx.metallib from the Metal kernel sources
- Build the
SwiftLM binary in release mode
Then start the server (models download automatically if not cached):
.build/release/SwiftLM \
--model mlx-community/gemma-4-26b-a4b-it-4bit \
--port 5413
(Add --stream-experts when running oversized MoE models to bypass macOS virtual memory swapping and stream expert layers directly from NVMe SSD.)
Run the Tests
make test
That is the supported entry point. It is a thin wrapper over:
./scripts/bootstrap_local_tests.sh # build the harness, make mlx.metallib findable
swift test --skip-build
Do not trust a bare swift test on a fresh checkout. MLX aborts the test process when it
cannot load its Metal library, and swift test can still exit 0 after printing
Failed to load the default metallib — on the pre-6.4 layout that happens at the second
test, so the run looks like a short green pass with almost nothing executed. make test
runs the bootstrap and then fails if fewer than 200 tests executed, so this failure mode
is reported instead of hidden.
The bootstrap handles both toolchains automatically:
- Swift 6.4+ (Xcode 27) — SwiftPM compiles the Cmlx kernels and embeds
default.metallib
in each test bundle, so nothing is copied. It only checks the Metal Toolchain is installed
(xcodebuild -downloadComponent MetalToolchain if it is not). Copying a metallib into
<bundle>.xctest/Contents/MacOS here invalidates the bundle signature and breaks the next
incremental build at CodeSign.
- Swift 6.2 (Xcode 26, the CI runner) — the metallib is not embedded; the bootstrap
fetches
mlx.metallib from the mlx Python wheel and copies it next to the test binaries.
To run a subset, make test-filter FILTER=PromptCacheTests (or pass swift test arguments
to scripts/run-tests.sh directly).
📊 Performance: Mac mini M6 (base, 32 GB)
The first SwiftLM numbers from a 32 GB Mac. Every other table in this README comes from 64 GB hardware. Running the same context lengths on this M6 exposed memory bugs that 64 GB machines had been hiding, and they are fixed in this release (see What 32 GB exposed).
Hardware: Mac mini (Mac18,5), Apple M6, 12-core GPU, 32 GB unified memory (170 GB/s), macOS 27.0. Metal working set 26.8 GB.
Method: scripts/profiling/m6_bench.py. One warm-up, then the median of 3 runs (1 run at 32K and above), temperature 0. Every prompt starts with a unique nonce, so the prompt cache can't hit, and hides a code word that the answer must return. A memory guard aborts any case whose swap grows by more than 2 GB. Raw results: docs/profiling/m6/.
Build: numbers are from the official release b782 binary unless marked † (earlier build, not re-run on b782).
What runs well on a 32 GB M6
| Model (4-bit unless noted) | Weights | Mode | Decode, short prompt | Longest prompt that passed | Peak GPU |
|---|
gemma-4-26b-a4b-it-4bit (MoE, ~4B active) | 15.3 GB | GPU | 53.9 tok/s | 80.7K tokens † | 19.5 GB † |
Qwen3.6-35B-A3B-UD-MLX-4bit (MoE, ~3B active) | 21.6 GB | GPU | 48.4 tok/s | 40.8K tokens ‡ | 22.1 GB |
Qwen3.6-35B-A3B-UD-MLX-4bit | 21.6 GB | --stream-experts | 14.1 tok/s | 40.8K tokens ‡ | 6.8 GB |
Qwen3.8-27B-4bit (dense) | 11.3 GB | GPU | 9.5 tok/s | 40.8K tokens † | 18.4 GB † |
gemma-4-26b-a4b-it-8bit | ~26 GB | GPU | swaps (+3.1 GB on the first prompt) † | — | — |
gemma-4-26b-a4b-it-8bit | ~26 GB | --stream-experts | 9.7 tok/s | 9.5K tokens (32K swapped †) | 7.1 GB |
‡ Throughput only: at 40.8K the needle check isn't reliable in either mode (see the Qwen3.6 table).
- MoE models are the sweet spot at 32 GB. Only the active experts are read for each token, so they decode 5–6× faster than a dense 27B. A 4-bit MoE with up to about 22 GB of weights runs entirely on the GPU.
- Qwen3.6-35B-A3B on a base M6 reaches 78% of the M1 Ultra 64 GB decode speed below (48.4 vs 61.7 tok/s). The M1 Ultra runs used temperature 0.6 and
--repeat-penalty 1.1; the M6 runs use temperature 0.
- Dense 27B decode is bandwidth-bound. 9.5 tok/s × 11.3 GB is about 107 GB/s, roughly 63% of the M6's rated 170 GB/s.
- An 8-bit 26 GB model needs SSD streaming and tops out at about 10K tokens of context.
Gemma-4-26B-A4B 4-bit — by prompt length
| Prompt tokens | Prefill / decode (tok/s) | TTFT | Peak GPU · swap growth |
|---|
| ~530 | 781 / 53.9 | 0.7 s | 14.3 GB · 0 |
| ~2.3K † | 963 / 50.2 | 2.5 s | 15.0 GB · 0 |
| ~9.5K † | 959 / 45.1 | 10.1 s | 15.8 GB · 0 |
| ~39.7K † | 757 / 31.0 | 53.3 s | 17.9 GB · 0 |
| ~80.7K † | 622 / 24.3 | 131.3 s | 19.5 GB · 0 |
This is the beginning of the README. Read the rest at the original. Read the full README ↗
