lok8s

 

Open-sourced in 2026 after ~9 years running real production workloads — new to GitHub, but shaped by nearly a decade of operating clusters for keeps.

 

✨ Why lok8s

• 🧰 Standard tools, not a walled garden. Targets are plain kustomize builds; Helm charts inflate via the khelm kustomize plugin (no helm CLI to install). The lo CLI and the folder convention are orchestration and ergonomics — the artifacts underneath are vanilla Kubernetes YAML you could kubectl apply by hand. No lock-in: lok8s produces standard manifests you can take anywhere.
• 💻 Dev-first, runs against any cluster. The default experience is a local kind cluster with TLS, a registry mirror, and a Tilt hot-reload loop that just works. lo build emits portable artifacts.yaml you can kubectl apply to any cluster. The Hetzner/KubeOne/CAPI provisioners are a convenience for standing up production — not a requirement.
• 🤖 AI built in, local-first. lo chat is an on-device assistant for your cluster (read-only by default, with a code-enforced safety gate), and lo mcp exposes every lo command to agents like Claude Code over MCP. No data leaves your machine unless you explicitly opt into a frontier model.
• 🧪 Nine years in production. The conventions here aren't speculative — they're the residue of ~9 years of running this in production, keeping what survived contact with reality and dropping what didn't.
• 🐚 Transparent and debuggable. lo is one static Go binary, but it still runs the same kubectl/kustomize/kind commands you'd run by hand: orchestrated, not reimplemented. lo --verbose shows every command, and the bash implementation the binary was ported from ships frozen inside every project (a project's lok8s.yaml can route any command to it), so you can always read, lint and step through the reference. (Why a Go binary, and why the bash stays)

 

🧭 Philosophy

A few principles shape every decision in lok8s:

1. Production is the reference; local dev is a nerfed overlay of it. You don't learn one workflow for your laptop and a different one for prod — it's the same tree, the same commands, the same cluster.lok8s.yaml. The local cluster is just production with the expensive parts swapped out.
2. One cluster = one folder, keyed by FQDN. Everything about a cluster lives under clusters/<fqdn>/. The domain is the identity, which makes multi-cluster and multi-environment setups obvious rather than clever.
3. Stand on standard tools. lok8s orchestrates kustomize, Helm (via khelm), kind, and Tilt — it doesn't reimplement them. If you know those, you already know most of lok8s.
4. Two concerns, kept apart. Cluster creation (a pluggable driver) is separate from cluster content (kustomize targets). Swapping how a cluster is born never touches what runs on it.
5. Minimal magic, maximal transparency. Rendered artifacts are plain YAML, every external call is a tool you could run yourself (lo --verbose prints it), and the only ordering primitive is an explicit spec.bootstrap list. When something breaks, you can see exactly what ran.

 

✅ When to use lok8s (and when not to)

Honesty up front — lok8s is opinionated, and that won't fit everyone.

Reach for lok8s when you want to…

• Have the same workflow from local kind to production instead of maintaining two parallel setups.
• Manage one or many clusters with a clear, FQDN-keyed folder convention.
• Keep a fast local dev loop (Tilt hot-reload, local registry, working TLS) without bespoke glue.
• Use kustomize + Helm charts as your manifest layer and want ergonomics around them.
• Provision on Hetzner (kind/KubeOne/CAPI) with sensible, batteries-included defaults — or just deploy to a cluster you already have.

Look elsewhere (or use lok8s only for the deploy side) when…

• You're fully invested in a managed platform's native workflow (EKS/GKE/AKS + their tooling) and don't want another convention on top.
• You prefer a pure-GitOps, controller-driven model (Argo/Flux as the source of truth) — lok8s can emit manifests for that, but its dev-loop is CLI/Tilt-centric, and the in-tree lo gitops layer is still being built.
• You need ready-made production provisioning on a cloud other than Hetzner today — the provisioning drivers currently target Hetzner. (You can still deploy lok8s-built artifacts to any cluster; you'd just bring your own provisioning.)
• A tool with no bash anywhere is a hard requirement for your team. lo is a single binary, but a project still carries the framework tree (.lok8s/), the Hetzner provider plugin is bash, and bash ≥ 4.3 is a prerequisite.

 

📦 Install

lo is a single static binary (linux/darwin × amd64/arm64), attached to every GitHub release with a checksums.txt. Download, verify, then run — nothing here is piped into a shell:

curl -fsSLO https://github.com/kernpilot/lok8s/releases/latest/download/lo-install.sh
curl -fsSLO https://github.com/kernpilot/lok8s/releases/latest/download/checksums.txt
sha256sum --ignore-missing -c checksums.txt   # macOS: shasum -a 256 --ignore-missing -c checksums.txt
less lo-install.sh                            # read it first
bash lo-install.sh                            # → ~/.local/bin/lo  (--dir, --version, --full, --dry-run)

The installer fetches lo-<os>-<arch>.tar.gz and checksums.txt from the release, refuses to extract anything whose SHA-256 does not match, and only then installs lo. What changed in this release and how to upgrade an existing project: v0.3.0 release notes. Prefer no script at all? The same four steps by hand:

V=v0.12.0; A=lo-linux-amd64.tar.gz             # your tag and platform
curl -fsSLO "https://github.com/kernpilot/lok8s/releases/download/${V}/${A}"
curl -fsSLO "https://github.com/kernpilot/lok8s/releases/download/${V}/checksums.txt"
sha256sum --ignore-missing -c checksums.txt
tar -xzf "${A}" lo && install -m 0755 lo ~/.local/bin/lo

Two builds, one tree. lo (the default, ~50 MB) is the core build: it runs the manifest render through the pinned kustomize binary and the two exec generators (khelm's ChartRenderer, the secrets.lok8s.dev Secret plugin) that lo toolchain install installs into the project with b. lo-full (bash lo-install.sh --full, ~120 MB) links the same kustomize API and khelm release into the binary and renders in-process — no kustomize, khelm or .kustomize/ needed. Both install as lo; lo --version names the build ((core) / (full)), and both render byte-identical output — that is the gate, see the Go binary reference.

Then scaffold a project: lo init project writes clusters/, lok8s.yaml, the .gitignore entries and a mise.toml (files only, no network). lo toolchain install writes .bin/b.yaml, installs b itself into .bin/ (pinned release, SHA-256-verified, no curl | sh) and runs b install for the pinned toolchain (kubectl, kustomize, khelm, the Secret plugin; kind, Tilt, mkcert for the dev loop). The framework assets a cluster references are embedded in the binary and ejected into .lok8s/ on first use:

mkdir my-project && cd my-project
lo init project                # the project files
lo toolchain install           # b + the pinned toolchain into .bin/
lo toolchain doctor            # b, kustomize, khelm and the Secret plugin at the pins

Joining a project that already has .bin/b.yaml? lo toolchain install never overwrites it — it prints a diff against the pins this lo was built with — and b install reproduces the committed b.lock. The full profile-based path (b env add github.com/kernpilot/lok8s#local && b install, which also syncs the framework tree and the frozen bash reference) is documented in The Toolchain.

Prefer mise? A mise.toml ships at the repo root — mise install && mise activate provisions the same toolchain. Then lo doctor to verify.

Cloning the repo directly? The argsh runtime is vendored in .bin/, so lo doctor runs immediately and tells you which tools are still missing — no b install needed just to diagnose the environment. make build produces the Go binary at bin/lo.

Legacy (argsh) install. New installs use lo-install.sh (above; also served at lok8s.io/lo-install.sh). Before the Go binary, a self-contained argsh script (lo-up) bootstrapped a project — installing b, the profile and the toolchain in one go. It does not install the Go binary. It is retired, not removed: the source and build live under .archive/legacy/install/ and the published bundle stays at lok8s.io/lo-up for existing users. Download and read it before running it: curl -fsSL https://lok8s.io/lo-up -o lo-up && less lo-up && sh lo-up.

Profiles — each ships only the binaries it needs:

Profile	Adds	Use case
core	framework only	Remote deploy only — no kind/Tilt
kustomize	kustomize plugins	Standalone kustomize plugin use
local	kind, Tilt, mkcert	Local dev (recommended starting point)
capi	clusterctl, hcloud	Cluster API provisioning
kubeone	kubeone, hcloud	KubeOne provisioning

Every profile ships a preconfigured clusters/lok8s.dev/ — a local cluster with working TLS out of the box. Bring your own FQDN later, or use *.[N].lok8s.dev for multiple projects.

Prerequisites: Docker, and bash ≥ 4.3 (macOS ships 3.2 — brew install bash). Everything else comes from b (or mise). Run lo doctor to check.

 

🐾 Quick Start

From zero to a running local cluster with a live dev loop:

lo use lok8s.dev          # select the active domain (ships preconfigured)
lo up                     # create the kind cluster, bootstrap infra, start Tilt
                          # → Tilt UI on the URL it prints (per-domain port, 10351–10499)
lo status                 # Running ✓
lo down                   # tear it all down when you're done

That's the interactive loop. For headless/CI or deploying to a remote cluster, the same definitions drive a build → deploy pipeline:

lo build                  # render the domain kustomization → clusters/<domain>/artifacts.yaml
lo deploy                 # apply built artifacts (CRDs first, then resources, with health waits)
lo lint                   # validate specs, bootstrap entries, and target references

lo build's output is plain Kubernetes YAML — kubectl apply -f clusters/<domain>/artifacts.yaml works against any cluster, with or without the rest of lok8s. With spec.build.artifacts: split the build additionally emits committable per-resource files under clusters/<domain>/artifacts/ (Secrets sops-encrypted) for GitOps consumers — see specs.

 

🧠 How it works

lok8s keeps two concerns strictly separate:

1. Cluster creation — how the cluster comes to exist. Handled by a driver (.lok8s/drivers/<kind>/main), selected by the kind: of your cluster.lok8s.yaml.
2. Cluster content — what runs on it. Plain kustomize, split into two planes: ordered bootstrap infrastructure and independent workload targets.

Everything is keyed by FQDN. A cluster domain owns a cluster (cluster.lok8s.yaml); a deployment domain ships content to another domain's cluster (deploy.lok8s.yaml).

flowchart LR
    spec["cluster.lok8s.yaml<br/>kind: Lo · KubeOne · Capi · Kkp"] --> drv{{driver}}
    drv -->|provision| k8s[("Kubernetes cluster")]
    k8s --> a["Plane A — bootstrap<br/>spec.bootstrap addons<br/>(CNI → LB → cert-manager …)"]
    a --> b["Plane B — workloads<br/>targets/* · kustomize"]
    b -->|"Tilt (dev) / lo deploy (CI)"| run(["running cluster"])

Concretely, lo up runs:

lo up                    # domain comes from `lo use` / --domain
 ├─ provision   driver creates the cluster        (kind / KubeOne / CAPI / KKP)
 ├─ bootstrap   framework applies spec.bootstrap   (CNI → MetalLB → cert-manager → …,
 │              addons in order, waits healthy      health-gated between stages)
 └─ tilt up     Tilt builds & live-reloads          (or: lo build + lo deploy, headless)

The two content planes:

• Plane A — bootstrap (cluster infrastructure). An ordered spec.bootstrap list of framework addons (CNI, load balancer, cert-manager, …) applied at provision time, with health waits between stages. The cluster isn't "ready" until this finishes.
• Plane B — workloads. User-named kustomize directories under targets/, each built independently into its own artifacts.yaml. No framework-level ordering — you express any ordering you need with Tilt's resource_deps or your GitOps engine.

Drivers are pluggable cluster backends:

Kind	Runtime	Use for
Lo	Docker + kind	Local dev / CI
KubeOne	KubeOne (Hetzner)	Self-managed production
Capi	Cluster API (Hetzner / CAPH today)	Declarative production provisioning
Kkp	Kubermatic KKP	Hosted control planes

The four drivers share one .lok8s/ tree and one spec format. Lo (local) needs nothing but Docker; the production drivers target Hetzner today but are optional — you can run lok8s entirely against clusters you provision yourself. Adding a driver for another backend is a documented extension point (see Extensibility).

For the full model — layer map, atoms/molecules, build/deploy pipeline — see ARCHITECTURE.md and the Concepts guide.

 

🗂️ The clusters/<fqdn>/ convention

This single convention is what makes the "same workflow everywhere" promise hold. The framework (/.lok8s/) stays flat and framework-owned; your content lives in a parallel clusters/ tree, one directory per cluster, named by its FQDN:

clusters/
├── lok8s.dev/                  # local dev cluster (ships with lok8s)
│   ├── cluster.lok8s.yaml       #   the spec — kind, bootstrap, network, …
│   ├── targets/                 #   workload plane: one kustomize dir per target
│   │   ├── platform/
│   │   └── apps/
│   ├── artifacts/               #   rendered output (gitignored, rebuilt on demand)
│   └── secrets/                 #   per-domain secret store (encrypted, opt-in)
├── cluster.example.in.net/     # production cluster (same structure!)
│   └── cluster.lok8s.yaml
└── api.example.com/            # deployment domain → deploys onto another cluster
    └── deploy.lok8s.yaml

Why this matters:

• The folder name is the cluster's API hostname, so the same brand can be served by different environments without collisions (clusters/example.com/ locally, clusters/cluster.example.in.net/ in prod).
• Secrets are per-domain (clusters/<domain>/secrets/), so dev and prod can never accidentally cross-pollinate credentials.
• A Deploy domain carries no cluster of its own — it clusterRefs another domain and ships workloads there, which is how one repo can target many clusters.
• It's just folders and YAML — diff-able, reviewable, and obvious to a newcomer.

See ARCHITECTURE.md for the complete tree and Specs reference for every field.

 

💻 Local development

The local experience is the part lok8s polishes hardest, because it's where you live day to day:

• One command, full loop. lo up creates a kind cluster, applies your spec.bootstrap infrastructure, and starts Tilt — which reads your services.yaml, builds images, wires docker_build + live_update, and gives you a UI at the URL it prints (a per-domain port in 10351–10499, so parallel projects never collide). Edit code → Tilt syncs and reloads. (Local Dev guide)
• Working TLS, no manual cert juggling. The secrets.lok8s.dev kustomize plugin's cert: generator mints leaf certificates from a shared local dev CA — no mkcert dance per project. lo trust adds the CA to your system store so browsers are happy. (Secrets guide, Kustomize plugins)
• Fast, shared registry mirrors. A pull-through mirror network can be shared across all your lok8s projects, so images are pulled once, not once-per-cluster. Opt in per project with spec.registries.shared.enabled: true (off by default — the safer single-network topology). (Shared Registries guide)
• Multi-project friendly. Use *.[N].lok8s.dev slots to run several local clusters on isolated Docker networks side by side.

Define services once, in a committed services.yaml, with personal overrides in a gitignored services.<config>.yaml ("I'm not working on the frontend today" → enabled: false). Each buildable service carries a small lok8s.yaml describing its build, ports, and live-update rules. (Services guide)

 

🚀 Production & deploying anywhere

Two honest paths to production:

1. Provision with lok8s (Hetzner today). The KubeOne and Capi drivers stand up real clusters on Hetzner Cloud (and bare metal via Hetzner Robot), with batteries-included networking, CNI, encryption-at-rest, and backups guidance. (CAPI · Bare Metal · Networking · Security · Backups)
2. Bring your own cluster. lo build renders standard artifacts.yaml — plain manifests you kubectl apply to whatever cluster your KUBECONFIG points at. EKS, GKE, a Raspberry Pi, a colleague's kind cluster — if kubectl can reach it, lok8s' output runs on it. (lo deploy automates the apply against the kubeconfig it resolves for the domain.)

The optional operator (shell-operator-based) reconciles Lo and Capi CRDs on a management cluster; its hooks are the same lo binary (lo operator <hook>), so cluster lifecycle can be declarative when you want it. (Operator guide)

The hosted, managed-platform layer (kubehz) is a separate product built on top of lok8s — lok8s works fully without it. No-lock-in is a design goal, not a slogan.

 

🤖 AI, built in

lok8s treats AI as a first-class, local-first capability — not a cloud dependency.

• lo chat — an on-device cluster assistant. Ask "why won't this deploy?" or "what's the LB IP?" and it routes through lo tools, gathers facts, and streams a markdown answer in your terminal. It runs read-only by default, enforced in code (not by trusting the model), so it can't mutate your cluster unless you switch posture with /posture open. Backends are local: Ollama or any OpenAI-compatible server (llama-server, llamafile, vLLM). Frontier CLIs (claude/gemini/codex) are strictly opt-in handoffs. Run lo chat --check for a guided setup. (Local AI guide)
• lo mcp — your CLI as agent tools. Every leaf lo command is exposed as an MCP tool (lo_status, lo_build, lo_deploy, …) over stdio, so agents like Claude Code or Cursor can drive lok8s the same way you do. Commands are tagged @readonly / @idempotent / @destructive, and a deterministic posture gate decides what an agent may actually run. A ready-to-use .mcp.json ships in the repo root. It launches bin/lo mcp start with the full tool surface enabled (see below).
• lo ai — wire skills into your assistant. The repo ships curated skills (cluster specs, services, addons, secrets, the dev loop, troubleshooting…). lo ai link claude symlinks them into .claude/skills/ for native loading; other agents get them by injection. lo ai check reports the whole setup at a glance.

Try it in two commands:

lo chat --check    # guided: checks the bridge + a local model, prints setup hints
lo chat            # then ask, e.g. "why won't my deployment start?"

If a piece needs setup, lo ai check / lo doctor tell you exactly what to run. (lo mcp is native to the binary. lo mcp claude|vscode|cursor enable writes the editor config. The bash variant, the argsh mcp builtin, starts as .lok8s/lo mcp from a checkout and wants argsh builtins install; see lo mcp.)

 

🧩 Extensibility

lok8s is conventions, not a cage — every layer has a documented seam:

• Custom cluster drivers. The built-in drivers are Go packages under internal/driver/. The bash driver contract, a file at .lok8s/drivers/<kind>/main implementing driver::provision, driver::destroy, driver::status, driver::kubeconfig (plus an optional driver::post_provision), is still honoured by the frozen implementation (a lok8s.yaml that routes provision to bash) and by lo drivers <kind> …; the binary's own dispatch reads only the Go registry today. The Driver Contract reference includes a complete worked example (a k3s driver).
• Your own addons. Drop a kustomize-buildable directory at .lok8s/addons/<name>/ (a khelm ChartRenderer + layered values.<driver>.yaml/values.<provider>.yaml, or any plain kustomization) and reference it by name in spec.bootstrap. (Addons guide)
• Adopt what you already have. Point a target's kustomization.yaml at an existing kustomize base (resources: [ ../path/to/your/kustomization ]), or inflate an existing Helm chart via a khelm ChartRenderer — no rewrite required.
• Add tools. The toolchain is managed by b: add an entry to .bin/b.yaml, assign it a profile group, b install, and it's on PATH.

 

🐚 Why a Go binary — and why the bash stays

For most of its life the lo CLI was bash, built on argsh. It is now a single static Go binary (cmd/lo, internal/), and the two facts below are both true on purpose:

• The binary orchestrates the same tools; it does not reimplement them. lok8s' job is to drive kubectl, kustomize, kind, clusterctl, kubeone, and friends. The Go code calls them as subprocesses through one seam, with the exact argv the bash used — no SDK drift, no version-matrix games. What lok8s runs is still what you'd run by hand, and lo --verbose still prints it. What Go buys is a one-file install with a verified checksum, a hermetic unit-test suite, and one --help/MCP schema that cannot drift from the code.
• The bash implementation is frozen, not deleted. It ships in every project under .lok8s/ as the reference the binary was ported from: a project's lok8s.yaml (spec.implementation) routes any command to it, ten differential parity harnesses diff the two implementations byte-for-byte in CI, and a go test fails if the command trees drift. When something looks wrong, you can still read the library under .lok8s/libs/, set -x, and step through it. If the bash is right and the binary is wrong, that is a bug we want reported. Full map, seams and the short list of deliberate differences: The Go lo binary.

argsh is still in the toolchain: the Hetzner provider plugin and the frozen tree source its runtime, and it is shellcheck-clean in CI (no new warnings allowed) alongside the Go lint. The kustomize plugins and the lo chat engine were Go from the start.

 

🔧 CLI Reference

Command	Description
lo use [domain]	Set / show the active domain
lo up [--open-tilt]	Provision cluster + bootstrap + start Tilt (--ci [--timeout <d>]: headless tilt ci, real exit status)
lo down	Stop Tilt + delete the cluster
lo status	Cluster health (the driver's status check)
lo provision	Provision cluster infra + apply spec.bootstrap addons (no Tilt, no targets/ deploy)
lo build	Render the domain kustomization → clusters/<domain>/artifacts.yaml
lo deploy [-l k=v]	Apply the domain artifact (CRDs → resources → health)
lo lint	Validate specs, bootstrap entries, target refs
lo doctor	Diagnose the local environment / toolchain
lo addons [name...] [--detail]	List / inspect framework bootstrap addons (--detail: the ones this cluster deploys, with category + configuration hints)
lo kubeconfig	Print the domain's kubeconfig (--oidc for the kubelogin exec-plugin)
lo destroy	Tear down a cluster
lo clean [--all]	Clean volumes; optionally prune Docker
lo chat	Local AI assistant (read-only by default)
lo ai check|skills|link|unlink	Manage AI skills + integration
lo mcp	Start the MCP tool server (stdio)
lo tilt up|down|status|restart|ci|preflight	Manage the Tilt environment (ci: headless build + deploy + wait; preflight: clear stuck-Terminating objects)
lo registry up|down|status|clean	Manage registry mirrors

Most commands act on the active domain (set by lo use) or an explicit --domain <domain> — in this table only lo use [domain] and lo addons [name...] take a positional. lo build and lo deploy take none: both act on the whole domain kustomization (deploy narrows via -l key=value, not targets). Global flags: --verbose|-v, --force|-f, --force-recreate, --remote|-r, --cluster|-s, --kubernetes, --config, --domain, --domain-sans. Full reference: docs/reference/cli.md.

 

📚 Documentation

Full documentation lives at lok8s.io. Good entry points:

• Getting Started · Concepts · Architecture
• Workflows: Local Dev · Services · Secrets · Deploying · Local AI
• Production: CAPI · Bare Metal · Networking · Operator
• Reference: CLI · Specs · Driver Contract · Kustomize Plugins

npm run docs:dev        # local docs site
npm run docs:build      # build

 

🤝 Contributing

Contributions are welcome — see CONTRIBUTING.md, the agent/contributor guide in AGENTS.md, and the test matrix in TESTING.md. In short: conventional commits, keep CI green (go test ./... + the parity harnesses + npm run lint + npm test), and security is paramount — never pipe untrusted remote content into a shell, never commit secrets, validate external input.

 

🔗 Related projects

lok8s is built on — and shares a philosophy with — a few sibling tools:

• b · binary.help — your one-stop binary manager. It installs and pins lok8s' toolchain, and lok8s ships as a b profile.
• argsh · arg.sh — the framework the original lo CLI was built on, and still the runtime of the frozen reference implementation and the provider plugins; it brings structure and maintainability to complex Bash (typed args, dispatch, generated --help).
• atty · atty.sh — a suckless-style PTY proxy (Zig) that drops an LLM exec dialog, atuin autosuggest, and guardrail confirmations between your terminal and your shell.

 

📜 License

MIT © 2025-present kernpilot