SuperWitness

Observability and evaluation for agent fleets. It shows what an agent did, how the product it ran on behaved, and whether the work was any good, joined on one run.

Website: superwitness.dev · Docs: docs.superwitness.dev · History: how it was built

SuperWitness is part of a family of self-hosted tools for working with agents. AgentPod manages the machines and runtimes agents live in; SuperPipeline owns boards, cards, runs and approval gates; SuperMessage is the Matrix client that puts agents in the conversation. SuperWitness records what happened and whether it was any good.

Status: v0.0.8 (changelog). Run documents cover SuperPipeline runs, with attempts read from the AgentPod hub, and a run registry lists the runs sources report; a web app reads and judges them. To try it without either product, SW_FAKE_SOURCES=1 serves one development run on loopback.

What it is for

Generic observability tools stop at a trace. SuperWitness goes one step further: from a trace to the run it belongs to, the attempts that executed it, the agent configuration that ran it, and the verdicts that judged it. That is what turns "did this configuration get better?" into a question you can answer.

It handles two kinds of data and keeps them apart:

	Telemetry	Verdicts
What	Traces, metrics, logs	Grader scores, offline eval results, calibration runs, configuration comparisons
Loss	Expected: sampled, short retention	Never
Where	Upstream engines (below)	SuperWitness's own database

Evidence that belongs to another product, such as a work run, a gate decision or a transcript, stays in that product. SuperWitness reads it through that product's API and never becomes its owner.

Rules this repository keeps

1. Meant to stand alone. The design rule is that no other SuperJackfruit product should be required, so that anyone's agents can send OTLP and record verdicts. This release is not there yet: it reads runs from SuperPipeline and attempts from AgentPod, and without them it runs only in fake mode (SW_FAKE_SOURCES=1).
2. Nothing requires it. Products send plain OTLP and never wait on SuperWitness.
3. Upstream engines, unforked. SuperWitness queries VictoriaTraces and VictoriaLogs, fed by the OpenTelemetry Collector, through their published query APIs only. It is designed to sit beside VictoriaMetrics and Perses the same way: upstream releases, run as released, never forked.
4. Permissive licences only in anything linked or shipped: MIT, Apache-2.0, BSD. No AGPL, so no Grafana, Loki, Tempo or Mimir. No source-available licences (ELv2, BSL, FSL, SSPL). Bundled components are credited in NOTICE.
5. Agents first. It is evaluation and observability for agent fleets, with product health underneath. It is not a general monitoring platform that also handles agents.

The docs site (docs-site/) and landing page (landing/) are checked against the code by internal/docsclaims: a page that names an SW_* variable, /v1 route or MCP tool the code does not define fails make test.

Install

curl -fsSL https://github.com/SuperJackfruitLabs/superwitness/releases/latest/download/install.sh | bash

This downloads the release tarball for your platform (Linux or macOS, amd64 or arm64), checks it against the release's SHA256SUMS, and installs superwitness and a short alias, suwi (a symlink to it), into ~/.local/bin. It never uses sudo, refuses to overwrite a different program at either name, and refuses a download it cannot verify. VERSION=v0.0.8 pins a release; BIN_DIR=/some/dir changes where it installs.

superwitness skill prints an operations skill for agents (what each command does and which ones change data); suwi skill install writes it to ~/.claude/skills, and SKILL_DIR=<dir> on the installer does the same. Needs v0.0.7 or later.

SuperWitness is a server. The installer sets up no service, writes no configuration and starts nothing; for running it as a server see Deploying and deploy/.

The installer and the macOS builds are published from v0.0.7 on; v0.0.6 and earlier have neither.

Running SuperWitness

make build
SW_FAKE_SOURCES=1 SW_DATABASE_URL=postgres://sw:[email protected]:5432/superwitness?sslmode=disable ./bin/superwitness
curl -H 'Authorization: Bearer dev:prn_human01:human' localhost:8790/v1/runs/superpipeline/brd_01/run_01

Fake mode serves one development run (brd_01/run_01) and accepts dev:<principal>:<human|agent|service> tokens. It refuses to listen on anything but loopback.

To try the run registry in fake mode, bind a development reporter and report a run:

SW_RUN_SOURCES=prn_reporter01=superpipeline SW_FAKE_SOURCES=1 SW_DATABASE_URL=… ./bin/superwitness &
curl -H 'Authorization: Bearer dev:prn_reporter01:service:runs:write' -d '{"source":"superpipeline",
  "external_ref":"brd_01/run_01","status":"running","source_status":"in_progress","reported_at":"2026-10-06T10:00:00Z"}' \
  localhost:8790/v1/runs
curl -H 'Authorization: Bearer dev:prn_human01:human' localhost:8790/v1/runs

A fourth segment of a development token lists its scopes, comma-separated.

To sign in to the app in fake mode, use the pieces make e2e uses: web/e2e/stub-hub.mjs stands in for AgentPod and web/e2e/serve.sh shows the settings.

The web app

The app in web/ is SvelteKit (Svelte 5, a static single-page build) on the vibekit design kit, which is vendored as a tarball in web/vendor/ (how to update it). It has four vibes (Daylight, Paper, Studio, Quiet), each in light and dark, and phone layouts. The person chooses under the account chip, Appearance.

make web         # npm ci, svelte-check, unit tests and the build, into internal/web/dist
make web-check   # the same, then fails if the committed internal/web/dist is not current

internal/web/dist is committed and embedded in the binary, so a change under web/src is committed together with the rebuilt bundle; CI runs make web-check and fails on a stale one. The server's Content-Security-Policy allows no inline script except the one SvelteKit emits in index.html: the server computes that script's hash at start-up and adds it to script-src.

make e2e builds the binary and runs Playwright against it in fake mode with a stand-in hub and a throwaway Postgres, so it needs Docker (run npx playwright install chromium once). On macOS with colima, point Docker at it first, for example export DOCKER_HOST=unix://$HOME/.colima/default/docker.sock. The sweep project visits every state of the app in every vibe, in light and dark, at desktop and phone widths, with axe checks and a watch for console errors and CSP violations. To run it alone (after the first full run has signed in): cd web && E2E_DATABASE_URL=… npx playwright test --project=sweep.

To look at the built app against stub data, with no Go server or database, run cd web && npm run build && node e2e/shots.mjs [path] [--open]; it screenshots a page in each vibe, light and dark, at 1440 and 390 px wide (the file's header lists its options).

Surface	Path
Run document	GET /v1/runs/superpipeline/{board}/{run}
Spans, logs (paged)	GET …/spans?cursor=&limit=, GET …/logs?cursor=&level=&limit=
Attempt → run	GET /v1/runs/by-attempt/{attempt} (302)
Record a verdict	POST /v1/verdicts
Report runs (service principals with runs:write)	POST /v1/runs
List runs	GET /v1/runs?source=&scope=&status=&executor=&since=&until=&needs_verdict=&cursor=&limit=
Rubrics	GET /v1/rubrics, GET /v1/rubrics/{id}/{version}
MCP (streamable HTTP)	/mcp: get_run, list_run_spans, list_run_logs, list_runs, record_verdict
The app (sign in with AgentPod)	/, /runs/superpipeline/{board}/{run}, /rubrics
Who am I, verdict history, scopes	GET /v1/me, GET /v1/verdicts?subject_kind=&subject_ref=, GET /v1/scopes
Health	GET /health (no auth; always 200 while alive, per-source status in the body)

Rubrics are append-only: superwitness rubric-add -id press -version 1 -name Press -scale '{"min":0,"max":1}' -body-file press.md -created-by prn_….

Deploying

The install guide covers one Linux host with systemd: the binary at /usr/local/bin/superwitness, deploy/superwitness.service, settings in /etc/superwitness/env from deploy/env.example, the AgentPod service credential's secret in /etc/superwitness/hub-client-secret (0600, the secret alone), and SuperWitness's own Postgres.

Staging and production (this project's own deploys)

Every push to main deploys that commit to staging (ci.yml deploy-staging, workbench, https://staging-app.superwitness.dev). A v* tag deploys production (release.yml deploy-production, https://app.superwitness.dev): after the release job publishes the tarballs and SHA256SUMS, it refuses unless the tag is on main and main's go, web and integration checks passed for that commit, then runs ssh root@$PROD_HOST superwitness <tag>. That key is forced to the host's prod-deploy wrapper (installed by the host's provisioning), which runs deploy/release.sh taken from the tag: download and verify the tarball, unpack it into /opt/superwitness/releases/<tag>, pg_dump the database (kept 14 days), migrate, apply the runtime grants below, point the unit at the release with a drop-in, wait for /health to report the tag, run deploy/staging/smoke.sh, and roll back to the previous release on any failure. The workflow ends with the same smoke against the public URL. Migrations are not rolled back; the dump is. RELEASE_CHECK=1 deploy/release.sh <tag> exercises everything up to the backup and changes nothing. The manual fallback is the host's documented ship-by-hand procedure.

Two database roles

Verdicts are append-only: triggers refuse UPDATE, DELETE and TRUNCATE on verdicts and rubrics. A table's owner can disable or drop those triggers, so the running service must not own the tables. SuperWitness takes two DSNs:

• SW_MIGRATE_DATABASE_URL connects as superwitness_owner, which owns the database and the tables. It is used only to run migrations.
• SW_DATABASE_URL connects as superwitness_app, the runtime role. It can insert and read verdicts and rubrics, insert, read and update registry runs, and nothing else.

If SW_MIGRATE_DATABASE_URL is unset, migrations run over SW_DATABASE_URL, which must then be the owner. That suits development only. SuperWitness never logs either DSN.

The split stops SQL-level misuse through the runtime pool, not a compromised process: when SW_MIGRATE_DATABASE_URL sits in the service env, the running process holds the owner password. To keep it out, leave SW_MIGRATE_DATABASE_URL only in the operator's env for superwitness migrate, run that before each upgrade, and also run GRANT SELECT ON goose_db_version TO superwitness_app; as the owner. serve then runs its background migration pass over SW_DATABASE_URL as the runtime role. With that grant, the pass succeeds when nothing is pending. Without it, the pass always fails. When a migration is pending, it fails either way, and the verdict store stays unavailable while the pass retries. TestRuntimeRoleMigrationPass checks all three cases.

Provisioning, run as the Postgres superuser before the first start. Passwords, or peer or certificate authentication, are set per deployment and are left out here:

CREATE ROLE superwitness_owner LOGIN;
CREATE ROLE superwitness_app LOGIN;
CREATE DATABASE superwitness OWNER superwitness_owner;
REVOKE ALL ON DATABASE superwitness FROM PUBLIC;
GRANT CONNECT ON DATABASE superwitness TO superwitness_app;

Then create the schema as the owner. superwitness migrate reads only SW_MIGRATE_DATABASE_URL (else SW_DATABASE_URL), runs every pending migration once, and exits 0, or non-zero with the error. It gives up after 60 s if Postgres does not answer:

set -a; . /etc/superwitness/env; set +a; superwitness migrate

Then grant the runtime role, connected to the superwitness database as superwitness_owner:

GRANT USAGE ON SCHEMA public TO superwitness_app;
GRANT SELECT, INSERT ON TABLE verdicts, rubrics TO superwitness_app;
GRANT SELECT, INSERT, UPDATE ON TABLE runs TO superwitness_app;
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLE sessions TO superwitness_app;

That is the whole runtime grant set. The tables have no sequences (keys are text), and the runtime pool never reads goose's goose_db_version table, so it needs no grant on it (unless serve migrates as the runtime role, above). Nothing is granted through ALTER DEFAULT PRIVILEGES: a later migration that adds a table must add that table's grant here. The runtime role cannot update, delete, truncate, alter or drop verdicts and rubrics or their triggers; it can update registry rows in runs but never delete them; sessions holds browser sign-ins and is the one table the runtime role may delete from; and it cannot create tables. TestRuntimeRoleCanAppendButNotRewrite and TestRuntimeRoleRegistryGrants run both blocks above verbatim against Postgres 17 and check all of this.

Upgrading: after superwitness migrate, run as the owner only the grants your starting version lacks. From 0.0.2, that is GRANT SELECT, INSERT, UPDATE, DELETE ON TABLE sessions TO superwitness_app;. From 0.0.1, it is both GRANT SELECT, INSERT, UPDATE ON TABLE runs TO superwitness_app; (added in 0.0.2) and the sessions grant (added in 0.0.3). Until the runs grant is in place the run registry answers 503 store_unavailable (verdicts are unaffected); until the sessions grant is, sign-in shows the "Can't reach the database" page.

On Postgres 14 or older, schema public belongs to the bootstrap superuser and every role may create tables in it. There, after the provision block and before superwitness migrate, run this as the superuser, connected to the superwitness database, so the owner role owns the schema and the runtime role cannot create tables. Postgres 15 and later need neither statement: public is owned by the database owner and PUBLIC has no CREATE on it.

ALTER SCHEMA public OWNER TO superwitness_owner;
REVOKE CREATE ON SCHEMA public FROM PUBLIC;

superwitness rubric-add follows the same split: it migrates over the migrate DSN and inserts over SW_DATABASE_URL.

Starting and health

superwitness serve connects its runtime pool with SW_DATABASE_URL only. It starts serving at once and runs its migrations in the background over the migrate DSN, retrying with backoff (1 s doubling to 30 s) while its Postgres is unreachable. Until they succeed, run documents show sources.verdicts: unavailable, POST /v1/verdicts returns 503 store_unavailable (retryable) and /health reports verdicts: unavailable.

/health stays 200 when a source is down, so an outside monitor can tell "SuperWitness is down" from "a source is down". A source outage shows in its body (sources.<name>) and in the superwitness.source.fetches{source,status} metric, which an alerting rule can watch.

Engine queries

SuperWitness uses only the engines' published query APIs. Lookback is 7 days.

• VictoriaTraces (Jaeger API), one request per service (agentpod-hub, agentpod-node-agent, superpipeline-api): GET /select/jaeger/api/traces?service=<svc>&tags={"run.id":"<run>"}&start=<µs>&end=<µs>&limit=100. These searches discover trace IDs. Each distinct trace is then fetched with GET /select/jaeger/api/traces/<traceID> (at most four concurrent lookups), so child spans from other services join even when they omit run.id. A failed or missing full trace makes the trace source report an error rather than returning a successful partial join.
• VictoriaLogs (LogsQL), POST /select/logsql/query, all queries prefixed _time:7d:
  • run filter ("run.id":="<run>" or trace_id:in("<trace>",…))
  • level filter (severity_number:range[17, 24] or severity_text:i("error")) (error shown; debug 5-8, info 9-12, warn 13-16)
  • page … | sort by (_time) | offset N | limit M
  • count … | stats count() as n
  • errors list: run filter, error level filter, | sort by (_time) | limit 50

Log fields read: severity_text, severity_number, trace_id, span_id, run.id, service.name.

Contributing and security

See CONTRIBUTING.md for how changes land and how to run the checks. Please report vulnerabilities privately, as described in SECURITY.md.

License

MIT.