posttrainllm

A Mac-first LLM factory + runtime + research substrate — and the browser playground that started the project. MIT-licensed, runs entirely locally, zero cloud.

The historical build phase is now organized as a guided, buildable learning lab: all 76 experiments have final dispositions, 18 retained techniques have recipe contracts, and nine ordered learning stages produce thirteen concrete artifacts—from a byte-level model and LoRA adapter to an eval ruler, local agent, MLX package, browser runtime receipt, and causal probe dossier.

The strongest measured claim: a 4B model distilled locally from ~99 frontier rollouts on one Mac matches DeepSeek-V4-pro at 100% on a multi-turn agentic tool-calling gate that Gemma-4-12b-qat scores 83% and the stock 4B 75% on. The full writeup, methodology, and head-to-head table is docs/learn/tool-calling-frontier-parity.md.

Live browser playground → · All experiments · Recipe lab · Learning paths · Buildable artifact contract · CLI lab reference · Mac CLI quickstart · Project closure and fresh-experiment gate · Frontier-parity result

---

What's actually in here

Surface	What it does	Where to look
Factory CLI (posttrainllm)	Data prep · post-training · eval gates · traces · packaging · serve/agent/runtime utilities. MLX-Swift on Apple Silicon. Retained as the runnable lab; no active target.	native-mac/, docs/NEXT.md
Factory docs	Retained sequence, run schema, eval protocol, packaging, and report shape.	docs/factory/
Mac app (SwiftUI)	GUI shell over the CLI. Parked except for a future minimal Factory Run Center.	native-mac/Sources/TinyGPTApp/, docs/parked/app-polish.md
Eval moat	E0 shared schema · BFCL · τ-bench · lm-eval (MLX-routed) · HumanEval (sandbox-exec) · eval-gate (CI).	docs/leaderboard.md, docs/research/mac_slm_leaderboard_v0.md
Agent runtime	OpenAI + Ollama-compatible local serve · multi-turn loop · FSM-constrained JSON · cloud-escalate · token-preserving .atraj trajectories.	docs/agent_runtime.md

---

Quickstart (Mac)

git clone https://github.com/PostTrainLLM/posttrainllm && cd posttrainllm/native-mac

# One-time: Metal toolchain (Xcode 27+).
xcodebuild -downloadComponent MetalToolchain

# Build with the currently selected full Xcode toolchain
DEVELOPER_DIR="$(xcode-select -p)" xcrun swift build -c release
BIN=.build/release/posttrainllm

# Try the CLI
$BIN --help
$BIN commands
$BIN help factory-run

# Train a tiny model from scratch on the bundled corpus (~2 min)
$BIN train --preset tiny --steps 200 \
  --corpus ../data/examples/tiny-corpus.txt --out /tmp/tiny.tinygpt

# Sample from it
$BIN sample /tmp/tiny.tinygpt --prompt "ROMEO:" --tokens 100

# Or LoRA-fine-tune a HuggingFace model on your data
$BIN sft <hf-model-dir-or-id> \
  --data your-corpus.jsonl --out my-adapter.tgla

# Export a distilled model or adapter for Python MLX / MLX-Swift
$BIN export-mlx /tmp/tiny.tinygpt --out /tmp/tiny-mlx
$BIN export-mlx my-adapter.tgla --out ./my-adapter-mlx

# Serve any posttrainllm or HF model on an OpenAI-compatible endpoint
$BIN serve <model> --port 8090
curl http://localhost:8090/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"posttrainllm","messages":[{"role":"user","content":"hi"}]}'

Mac app — same project, SwiftUI frontend:

DEVELOPER_DIR="$(xcode-select -p)" xcrun swift build --product TinyGPTApp
.build/debug/TinyGPTApp

---

Quickstart (browser playground)

Open posttrainllm.com.

• Load pretrained model — Shakespeare checkpoint, generate immediately.
• Train your own from scratch — ~15 min on the larger presets, WebGPU on Chrome/Edge 113+ and Safari 18+.

To build it locally:

bash wasm/build_wasm.sh          # needs Emscripten SDK
pnpm install
pnpm --dir browser run dev

---

Headline measured numbers

All on a single Apple M-series laptop (primary: M5 Pro / 48 GB). No multi-GPU, no cloud, no asterisk.

What	Number	Source
Multi-turn agentic hard gate, 4B distilled	100% (vs DeepSeek-V4-pro 100%, Gemma-4-12b-qat 83%, stock 4B+plan 75%)	docs/learn/tool-calling-frontier-parity.md §8.1–8.2
Distillation budget	~99 frontier rollouts · LoRA SFT · 16 layers · lr 1e-5 · 4 epochs	same
Decode throughput, flagship-huge-v5 (221M)	696 tok/s sustained	docs/research/mac_decode_baseline_m5pro.md
Decode, mega-pilot (960M)	293 tok/s	same
First-token latency (TTFT)	5.8 ms p99	same
Training step (Huge, B=8)	42 ms/step	same
ANE chain (Qwen3 28-block, layer-chunked Core ML)	17 tok/s	docs/PLAN.md §1
Browser WebGPU end-to-end vs WASM	10.67× on Large in a paired Apple M5 Pro ABBA run; 4.72% max final-loss drift; other presets remain historical	evals/verified-wins/webgpu-paired-result-v1.json
Largest browser-path model allocation	473M params via Memory64 (Node; in-browser Memory64 remains ABI-limited)	browser/src/pages/devlog.astro
WebGPU vs WASM correctness	Paired Large run passed the frozen <5% final-loss-drift gate with zero runtime errors

---

Honest scope

What this is:

• A single-developer project that ships in public, MIT-licensed.
• Mac-first (Apple Silicon, MLX-Swift). The Linux/CUDA path is not built. Browser path stays first-class.
• A factory: target → data → post-training → eval → package → report.

What this isn't (yet):

• An enterprise platform. There's no auth, no SSO, no team workspace, no SLA. Solo project.
• A frontier-model trainer. Specialists distilled from frontier teachers; the floor is wherever the teacher set it.
• A general-purpose agent platform (Vercel Eve / Cursor / Replit Agent). Different shape — these run on your hardware on a model you trained, not on someone's cloud on someone's model.

Known limits, named:

• Specialist track learned a depth tax: out-of-domain breadth regressed 60% → 42% on the catastrophic-forgetting gate during the v11 → v12 transition. We measure it instead of papering over it. See docs/learn/tool-calling-frontier-parity.md §8.4.
• The Apple on-device foundation model (4096-token ctx) can't ground actions — verdict: free routing floor, never a dependency. docs/learn/apple-on-device-foundation-models.md.
• Multi-GPU / distributed: a single-Mac data-parallel mlx.distributed PoC exists (scripts/archive/dist_dp_poc.py); multi-Mac is unblocked but unproven.

---

Repo layout

posttrainllm/
  native-mac/      MLX-Swift CLI + SwiftUI app — the main surface
  browser/         The original WebGPU playground (Astro + WGSL + WASM)
  webgpu/          Hand-written WGSL kernels (forward, backward, AdamW, FA2)
  wasm/            C++ kernels + reference C++ model → WebAssembly
  python_ref/      PyTorch reference: model, train, sample, LoRA, bench
  evals/           Smoke tests + eval fixtures (BFCL, MATH-500, etc.)
  scripts/         BFCL drivers, mlx.distributed PoC, leaderboard builders
  tests/           Correctness tests — finite-diff, overfit, parity
  configs/         Model / training / LoRA / PEFT settings as JSON
  grammars/        Pace system prompts + response JSON schemas
  data/            Dataset builder + example corpora
  specialists/     Trained-specialist registry + model cards
  benchmarks/      Published benchmark landing pages
  docs/            Plan + recipes + research + sessions + per-topic guides
  docs-site/       Blume publisher for docs/ (owns no content of its own)

Build, test, and lint entry points are in CONTRIBUTING.md.

The three web packages (browser/, webgpu/, wasm/) are one application: browser/ is the Astro shell and picks a backend at runtime, webgpu/ is the GPU path and wasm/ the CPU path. They are wired together by relative imports and browser/tsconfig.json, so they move together.

---

Docs (curated)

Start here:

• PROJECT_STATUS.md — completion truth and public-release boundary.
• docs/NEXT.md — closure receipt and fresh-experiment admission rule.
• docs/attempt-ledger.md — all 76 final experiment dispositions.
• docs/recipes/registry.json — all 18 retained recipe contracts.
• docs/learn/path-registry.json — nine ready learning paths, labs, and mastery gates.
• docs/learn/artifact-journey.json — nine stages and thirteen artifacts, each with build, modify, tune, prove, and package actions.
• docs/factory/ — run schema, eval protocol, packaging, and report templates.
• docs/parked/ — paused lanes retained for an explicitly reactivated target.
• docs/learn/tool-calling-frontier-parity.md — the strongest measured result, head-to-head methodology.
• docs/agent_runtime.md — the agent runtime including the B22 token-preserving trajectory format.
• — closed loop: rollouts → SFT JSONL → trained specialist.

Deeper:

• docs/performance/performance.md, docs/archive/lessons.md, docs/training/index.md, docs/techniques/distillation.md, docs/techniques/interpretability.md, docs/performance/determinism.md, docs/CITATIONS.md.

Sessions + research:

• docs/sessions/ — chronological retrospectives.
• docs/research/ — written-up investigations.
• docs/learn/ — primers + the project's learning map.

External projects to study

• oumi-ai/oumi — open-source ML training and evaluation platform
• google-research/tuning_playbook — a guide for hyperparameter tuning
• radixark/miles — a personal learning roadmap tool
• mvakde.github.io/blog/44-on-arc-1 — essay on ARC-AGI approaches

---

License

MIT — see LICENSE.

Author: Sarthak Agrawal (@sarthakagrawal927).

If posttrainllm is useful to your work or you want to chat about Mac-first ML training: open an issue, or reach out on LinkedIn / Twitter.