Scarab

A CI engine for Kubernetes — forge-native, on a durable core.

A CI system for Kubernetes with the forge pieces built in: webhooks, pipelines in your repo, PR checks, secrets, approvals and a UI. Each run is stored as durable state in Postgres, so when the control plane restarts, the run resumes instead of starting over. Written in Rust.

Live demo · Context · ADRs · Positioning · Run it locally

---

The dashboard: an action inbox of runs suspended on a gate, recently-visited repos as pass/fail status cards, and the recency-sorted repo list. ⌘K to jump anywhere.

CI tools fall into two groups, and neither does the whole job. Workflow engines (Argo, Tekton, Temporal) have a durable, crash-safe core, but they aren't CI: you build the forge integration, in-repo config, PR checks, identity, secrets, DSL and UI yourself. Forge-native CIs (GitHub Actions, GitLab, Woodpecker, Drone) give you all of that, but underneath they are job runners. If the control plane restarts mid-run, the run is orphaned or starts again from scratch.

Scarab is built to be both: a forge-native CI with the batteries included, on a control plane that stores each run as durable state instead of a fire-and-forget process. The engine is a crash-safe state machine on Postgres (the DBOS/Temporal pattern). When the control plane restarts, the run resumes from its last completed step, and each step runs exactly once. Self-hosted CIs like Woodpecker and Drone strand in-flight builds on a restart: they stay stuck in running and leave containers behind (woodpecker-ci#3427, drone#2189). A Scarab run picks up where it left off.

Durability is how the engine is built, not the main reason to pick it. Several useful features come from it: resuming a run, restarting a single step, approval gates that survive a restart, and an event log you can inspect. It matters most as pipelines get longer and more autonomous (agents, multi-day workflows, long review gates), where "the job died, click re-run" is no longer good enough. Scarab holds a suspended run at near-zero cost and resumes where it paused.

Lineage. Scarab's design owes a lot to Woodpecker, a lean, forge-native CI without enterprise overhead. Woodpecker's limits shaped it just as much: the many backends it has to support, and the pace a volunteer project can keep up. Supporting only Kubernetes drops the extra backends on purpose, and a small team building AI-first can now move faster than a volunteer project can.

Status

Scarab is not a shipped product yet. The durable engine is tested against real Postgres in CI. The forge and Kubernetes integrations are implemented and tested, some in CI and some by hand, but none of them has carried production load.

• Tested against real Postgres, in CI. The durable engine: crash-mid-run → resume, exactly-once step execution, restart-a-step, durable gates, content-addressed workspace, invoke inlining, scheduler (concurrency/fairness/supersede), secrets + fork-PR lockout, and a self-hosted OIDC issuer. .github/workflows/ci.yml runs the workspace suite against real Postgres and MinIO on every push to main and on every PR that touches Rust code. In CI a Postgres-backed test that would skip fails instead.
• Tested against a fake forge. The forge-native flow end-to-end: signed webhook → in-repo .scarab config → run → commit statuses posted back → OAuth-gated reads.
• Implemented, and run against real forges by hand. Two forge adapters, GitHub (/webhooks/github) and Forgejo (/webhooks/forgejo), each with HMAC webhook ingest and commit-status posting over real HTTP. just test e2e drives a real Forgejo, running in the kind cluster, through add-connection → bind → hook → signed push → Run. It doesn't check status posting. CI runs it only on PRs labelled ci:full and weekly. The live demo and this repo's own dogfood setup (just stack up dogfood, deploy/kind/) take their webhooks from a GitHub App. Each adapter also has a live contract test (tests/contract_live.rs) that you run by hand.
• Implemented, and tested on a kind cluster. Pod-per-step execution on Kubernetes (scarab-executor-k8s): re-attach after an executor relaunch, step deadlines, cancel, clone and build steps, the workspace feed and drain, artifact harvest, and the results-egress sidecar image (docker/sidecar/). just test e2e runs these live-cluster tests on the scarab-dev kind cluster (created if missing), and .github/workflows/e2e.yml runs the same recipe on PRs labelled ci:full and weekly, not on every PR.
• Tested for one case. Control-plane crash and resume with a real Pod in flight. The crash_resume test in crates/scarab-e2e SIGKILLs the server from the kind node while a step Pod runs; the kubelet restarts the container in the same Pod, and the new process re-adopts the Attempt without running it again. It passes in just test e2e, on a local kind cluster and in .github/workflows/e2e.yml. A server Pod moved to another node, or a lost node, is not tested.
• Nothing has run at production scale.

We build in thin end-to-end (tracer-bullet) slices. See docs/positioning.md for what we do and don't claim, and why.

Live demo

demo-scarab.ahiravan.dev runs the real thing: the control plane, the workspace service, and every step Pod on a single Oracle Always Free ARM box under k3s, reached through a Cloudflare Tunnel, with Cloudflare R2 as the object store, for $0/month. The deployment mode is deploy/demo-oracle/, and its README is the operator guide.

Sign in with GitHub and you land as Viewer. Viewer has no Write, so you can look around but you can't dispatch a run. The server has no anonymous read path. It's a demo box on free-tier capacity that can be reclaimed, so there is no SLA.

Documentation

The full docs site, docs.scarab, is built with Astro Starlight and published to GitHub Pages on tag. It has the getting-started guides, the pipeline authoring/config reference, the generated OpenAPI, and all 70+ ADRs. In-repo, the same sources live under:

• CONTEXT.md — the core idea, the durability contract, non-goals, and the ubiquitous language every crate/API/UI must use.
• docs/adr/ — 70+ Architecture Decision Records (why each major decision was made).
• docs/positioning.md — what we do and don't claim, and why.
• Issues — tracked in git-bug (embedded in this repo): git-bug bug to list, git-bug termui for the TUI.

The run view

A run is durable state: the step DAG, live logs and an append-only event timeline. Here a flaky step was retried automatically, then the run paused on a manual deploy-prod gate, where it waits at near-zero cost until someone approves.

What makes it different

Forge CI and engine together	A forge-native CI (DSL, secrets, approvals, identity, UI) on a durable engine. You don't have to build a CI on top of a workflow engine, or accept a job runner with nothing durable underneath.
Runs are state, not processes	A DBOS-pattern durable state machine on Postgres. When the control plane restarts, the run resumes from its last completed step, and each step runs exactly once.
Keyless identity	Forge-agnostic OIDC, with Scarab itself as an OIDC issuer for keyless federation to your cloud. (GitHub + Forgejo adapters implemented; the live-forge loop is not yet hardened; see Status.)
A real DSL	You write YAML with CEL expressions. Scarab compiles it to a typed IR, and the IR (not the YAML) defines what a valid pipeline is. Every pipeline is a flat recursive DAG: invoke handles reuse and matrix is a modifier.

Every other design decision gets checked against the durable core. The reason to use Scarab, though, is that the CI and the engine come as one piece. See docs/positioning.md for how we describe Scarab and what we don't claim yet.

Design at a glance

Aspect	Choice
Core	DBOS-pattern durable state machine on Postgres (Rust); object store for blobs
Execution	Pod-per-step, content-addressed workspace (per-file merkle CAS); k8s-only
Pipeline model	Flat recursive DAG (Pipeline → Step); invoke = reuse; matrix = modifier
DSL	Typed IR (the real DSL) + YAML frontend + CEL expressions
API	REST/OpenAPI + SSE (dogfooded by UI & CLI); roles coordinate through the Postgres outbox, and the Data Depot speaks HTTP
Code	Hexagonal, compiler-pure domain crates + per-vendor adapter crates; one converged binary
Security	Forge-agnostic OIDC identity; Scarab as OIDC issuer for keyless federation
UI	SolidJS + generated OpenAPI client

Workspace layout

CONTEXT.md            ubiquitous language + system overview
docs/adr/             architecture decision records
crates/
  scarab-engine       durable core (pure): DAG state machine, scheduler, ports
  scarab-pipeline     IR, YAML→IR, CEL, validation (pure)
  scarab-forge        ForgePort + canonical Event/Status (pure)
  scarab-identity     Authenticator, OidcIssuer, RBAC (pure)
  scarab-secrets      SecretProvider (pure)
  scarab-storage      ObjectStore + Cas (pure)
  scarab-project      Org/Repo/Project/Environment (pure)
  scarab-db-postgres · scarab-secrets-postgres · scarab-storage-s3   state/blob adapters
  scarab-executor-{k8s,local} · scarab-forge-{github,forgejo}        exec/forge adapters
  scarab-testkit      fakes: FakeClock / InMemoryDb / FakeExecutor
  scarab-server       composition root: axum + OpenAPI + SSE; role runner
  scarab-cli          CLI over the same REST API as the UI

Building

cargo check --workspace
cargo run -p scarab-server -- --help

Local dev

The local stack's host address lives in deploy/kind/.env (gitignored, seeded from the committed deploy/kind/.env.example the first time you run a recipe): SCARAB_ADDR is where the API and UI answer, SCARAB_API_URL is where just ui proxies. Real secrets for the dogfood stack live in ~/.config/scarab/dogfood.env (see deploy/kind/README.md).

just ui            # Vite dev server on http://localhost:5173, proxies /v1 → the stack
just stack reload  # rebuild scarab-server and roll the stack's Pods onto it

Auth is disabled in the dev stack (scarab.devInsecure), so every request is an Owner.

Full dev stack (k8s)

One command builds scarab-server on your machine, creates a kind cluster with Postgres and MinIO in it, and installs the Helm chart with the server and the Depot running that binary:

just stack up      # kind + Postgres + MinIO + the chart; the server binary is built on the host
just stack demo    # POST a one-step pipeline, poll until it succeeds, print logs
just stack reload  # rebuild and roll the server onto the new binary (no image build)
just stack down    # delete the cluster (`just stack down keep` keeps its data)

Requires docker, kind, kubectl, helm, cargo, and python3, plus zig and cargo-zigbuild on macOS (the binary is cross-built for Linux). The API and UI answer on SCARAB_ADDR from deploy/kind/.env. The cluster's kubeconfig is deploy/kind/.kubeconfig, checked against the cluster before every call, so the recipes never touch an ambient (e.g. production) context. just stack up dogfood is the same stack in GitHub App mode behind a Cloudflare tunnel. See deploy/kind/README.md.

just test e2e runs every live suite against the same release, switched to its e2e mode: its own database and bucket, so dev data is not touched, and a cloudflared quick tunnel in front of the in-cluster Forgejo. just stack up switches it back. just test runs the workspace suite against a separate compose Postgres and MinIO (deploy/test-services/), then the UI no-DOM suite. just with no arguments lists every recipe.

Without Docker/kind, the server still runs API-only against a local Postgres and a filesystem object store (--object-dir); the background driver is skipped until a cluster is reachable.

Running or testing Scarab locally: use the just recipes as the canonical entrypoints (just stack up/demo/down, just stack up dogfood, just test, just test e2e) rather than hand-rolled docker/kubectl/helm commands — they carry the right env, isolated kubeconfig, image source, and kube safety checks. Missing something? Add a recipe. (Agents: this is also in CLAUDE.md.)

API contract workflow

openapi.json (committed at the repo root) and the generated TS client (ui/scarab-web-ui/src/api/schema.ts) are checked for drift in CI: the build fails if either is out of date. After changing any route or DTO:

cargo run -p scarab-server -- --emit-openapi openapi.json
cd ui/scarab-web-ui && npm run gen && npm run typecheck

Full-route coverage is enforced by the test route_table_equals_the_openapi_spec (crates/scarab-server/tests/step_up_routes.rs): a registered route missing from the spec (no #[utoipa::path] annotation), or a spec operation with no route, fails the suite.

CI labels on pull requests

Actions minutes are limited, so two PR labels control what CI runs on a PR. Pushes to main and v* tags ignore both.

Label	Effect
none	ci runs the workspace suite and the other jobs its path filters select; image builds the images whose inputs changed. No cluster job runs.
ci:skip	Every job in ci, e2e and image skips without starting a runner. Adding it cancels a run in progress. Removing it starts a run.
ci:full	ci and image ignore the path filters and run every job, and e2e runs just test e2e (every live suite on a kind cluster; it makes network calls through a cloudflared quick tunnel). Adding it starts a run.

ci:skip wins when both are set. A PR with ci:skip never passes just pr <n>: remove the label, wait for the run, then run the gate.

For a stack of PRs, where only the tip's run matters:

1. Label every PR except the tip ci:skip.
2. Label the tip ci:full. Its path filters compare it with the PR below it, not with main, so without the label they would skip jobs that the lower PRs' changes need.
3. To merge, take the PRs bottom-up: remove ci:skip from the bottom PR, wait for its run, check it with just pr, merge, and repeat. To spend one run instead of one per PR, merge the stack down into the bottom PR's branch once the tip is green, then remove ci:skip from the bottom PR and merge it.

A change to any other label also starts a run, because GitHub cannot filter triggers by label name. In that run every job skips, it does not cancel a run in progress, and just pr ignores it when the commit has a real run.

License

GNU Affero General Public License v3.0 (AGPL-3.0-only). Running a modified Scarab as a network service obliges you to offer users its source.

References

• ADR-0001 — CI as durable execution
• ADR-0005 — Tenancy & deployment; Kubernetes as the only backend
• ADR-0040 — Documentation site: Astro Starlight, in-repo, DESIGN.md-branded
• ADR-0046 — Forge auth is adapter-internal; GitHub + Forgejo adapters in v1
• ADR-0054 — Product surface: embedded UI, run cancellation, API/CLI truthfulness