Apple LLM Performance Tracker

Which open-weight models actually run well on which Apple silicon, and through which engine.

Live site: https://dreamingwell.github.io/apple-llm-performance/

Pick a CPU model, a memory size and a machine count. The page then tells you, per model: whether it runs, which engine to use, which quantisation that engine should load at that memory size, how much fidelity you give up, and how many concurrent contexts fit in what's left.

Corrections and additions are welcome — open an issue or send a PR. Everything the page renders from is in this repo, in plain Python data structures.

This was made by AI, and AI contributions are welcome

The research, the data and the code here were produced by Claude working from published model cards, engine source trees and issue trackers, with a human directing it and pushing back. We are saying so up front because you should weigh the page accordingly: it is a well-sourced secondary compilation, not first-hand measurement, and every figure links to where it came from.

Because of that, we welcome reasonable AI-assisted contributions. Use whatever tools you like. What we ask is the same thing we ask of ourselves:

• Cite the source for every number — a repo, a model card, an issue, a commit.
• Don't invent a benchmark score, a file size or an issue number. If it can't be linked, leave it out and say it's unverified.
• Check that a quant actually loads on the engine you're claiming, rather than inferring support from a file existing.
• Keep it to a reviewable size. A PR touching one model is easy to check; a sweeping rewrite is not.

We are especially interested in new and useful models. If something has shipped open weights and runs on Apple silicon, we want it here — see CONTRIBUTING.md, or AGENTS.md if you are an agent.

Engines covered

Engine	Format	Interface	API
llama.cpp	GGUF	CLI + llama-server	OpenAI + Anthropic
Ollama	MLX on Apple silicon, GGUF elsewhere	background server + CLI	OpenAI + native
LM Studio	GGUF and MLX	desktop app + lms + server	OpenAI
oMLX	MLX	menu-bar app + server	OpenAI + Anthropic
vLLM Metal	MLX	CLI + the vLLM server	OpenAI
vllm-mlx	MLX	server	OpenAI + Anthropic
mlx-lm	MLX	CLI + mlx_lm.server	OpenAI (minimal)
DwarfStar / ds4	purpose-built GGUF	CLI + server + agent	OpenAI + Anthropic

Generative media

None of the servers above can load a diffusion or audio model, so these are a separate set. The page pairs models with engines of the same modality and never shows one against the other.

Engine	Output	Interface
mflux	image	CLI + Python
MLX-Audio	audio - TTS, STT, music	CLI + Python + web UI
MLX-Video	video, image	CLI + Python
DiffusionKit	image	CLI + Python

Every one of them speaks HTTP. None is desktop-only.

Building it

Standard library only. No install step, no dependencies.

python3 tracker/validate.py   # checks every record in data/
python3 tracker/build.py      # writes docs/index.html

Both run in CI on every push and pull request, so a contribution's shape is machine-checked before anyone reads it.

docs/ is generated and deliberately not committed — Pages serves the artifact that .github/workflows/deploy.yml builds on every push to main, so a merged PR is a deployment and nobody has to resolve conflicts in a 780 KB generated HTML file.

To refresh the tracked issue states (needs a token — the API allows 60 unauthenticated requests an hour and this makes ~117):

GITHUB_TOKEN=$(gh auth token) python3 tracker/probe.py > tracker/watch-state.txt

.github/workflows/refresh.yml does that twice a day and commits the result, which triggers a deploy. tracker/watch.sh is the same loop for running locally.

To re-measure one model's quant ladder from Hugging Face (this one hits the network):

python3 tracker/measure.py --model qwen38

It rewrites the LADDER block in that model's file and nothing else.

What's where

One record, one file. A model, an engine, a use case and an issue tracker each own exactly one file, so two people adding two models never touch the same file.

data/models/<id>.py       one model: identity, scores, per-engine status,
                          measured quant ladder, KV geometry
data/engines/<id>.py      one engine: what it is, its API, its cross-cutting issues
data/use_cases/<id>.py    one "What for?" category and its curated ranking
data/issues/<repo>.py     tracked issues for one upstream repository
data/pr_keys.py           which issue keys are pull requests

tracker/registry.py       loads data/ and assembles it; the only file that
                          knows the schema
tracker/validate.py       enforces the schema — CI runs this
tracker/render_status.py  the HTML template and the page's prose
tracker/bands.py          fidelity thresholds and per-model quant caveats
tracker/build.py          renders docs/index.html
tracker/measure.py        re-measures a quant ladder from the Hugging Face API
tracker/probe.py          refreshes watch-state.txt from the GitHub API
tracker/watch-state.txt   last polled state of every tracked issue
tools/check_output.py     sanity-checks the rendered page — CI runs this
assets/                   social card source and output
docs/                     GENERATED, gitignored — the built site

Start in data/models/ — that's where the facts are. Everything in tracker/ is plumbing around them.

If you are an AI agent working on this repository, read AGENTS.md first. It is the maintenance contract: where to look for new models, how to derive each figure, and the pitfalls that have already bitten us here.

How the numbers are derived

• Weights are summed file sizes from the linked Hugging Face repository — safetensors for MLX builds, GGUF for the rest. Measured, not estimated.
• Bits per weight is bytes * 8 / total parameters, so it is the effective figure rather than whatever the quant is named. Those diverge badly on MoE models: GLM-5.2's UD-IQ1_S is really 2.33 bpw, because the non-expert tensors are carried at higher precision.
• Expert-pruned builds (REAP and similar) carry no bits-per-weight number at all. Their loss is structural — whole experts deleted — and a bits figure would flatter them.
• Fidelity bands follow published evidence rather than opinion. Unsloth's Dynamic 3.0 notes state plainly that 1-bit builds should not be used for agentic work; their Qwen3.5 sweep puts 99.9% KL divergence at 0.41 for Q4_K_XL, 1.53 at IQ3_XXS, 2.91 at Q2_K_XL and 4.22 at IQ2_XXS.
• KV cost per token is derived from each model's published config.json, counting only layers whose cache grows with context. Sliding-window layers are bounded by the window and linear/Mamba/KDA layers hold a fixed recurrent state, so neither belongs in a per-token figure. Latent-attention models store one compressed vector per layer instead of separate K and V, which is why they are so much cheaper.
• Use-case rankings are curated, not computed. The benchmarks are not mutually comparable — Terminal-Bench 2.0 and 2.1 are different tests — so sorting by "score" would invent a comparison that does not exist. Each list is explicit, with the evidence for each placement, and the browser picks the highest-ranked model that actually fits at a usable precision.
• Fit assumes a 90% wired-memory limit plus framework overhead — ~10 GB for an LLM server holding a paged KV pool, ~1.5 GB for an image or audio runtime. It answers "does this load", not "does this run well".

Limits worth stating plainly

Nothing here has been benchmarked on this hardware by us. Benchmark scores are vendor- or aggregator-reported. Issue states are a twice-daily snapshot. Every memory and throughput figure is arithmetic over published specifications, not a measurement. Treat the page as a shortlist filter and verify anything you intend to spend money on.

If you have measured something on real hardware, that is the most valuable kind of contribution — see CONTRIBUTING.md.

Licence

MIT for the code. The benchmark figures and issue references belong to their respective projects and are cited on the page.