guiltyspark

guiltyspark is a deployable agent for watching Loki logs, spotting unknown problems, and turning the useful findings into actionable reports. It is designed to run on any always-on Docker host.

The core loop is intentionally simple:

1. Query Loki for recent logs.
2. Group noisy log lines into incidents.
3. Ask codex exec to explain likely bugs, misconfigurations, and improvement opportunities.
4. Store findings locally so repeated noise does not alert forever.
5. Associate the incident with its configured GitHub repository.
6. In an isolated clone, let Codex prepare a minimal fix and regression tests.
7. Enforce patch policy, run validation, and optionally open a PR.

What It Does

• Looks across any apps sending logs to Loki.
• Searches for errors, retries, auth failures, degraded services, suspicious restarts, slow requests, and repeated warnings.
• Summarizes evidence instead of forwarding raw log spam.
• Keeps SQLite state for cursors and duplicate suppression.
• Emits JSONL findings that can be tailed, shipped, or wired into notifications.
• Uses a ChatGPT-authenticated Codex session, so ChatGPT Pro can cover usage instead of a separately billed OpenAI API key.
• Runs on a Raspberry Pi with Docker Compose.

What It Does Not Do Yet

• It never pushes code unless the target is explicitly configured in draft-pr or pr mode.
• It does not assume this Codex chat controls the Pi.
• It does not require a local repo checkout unless you enable fix/PR workflows.

Quick Start

cp .env.example .env
# Set GUILTYSPARK_TARGETS_JSON in .env or in Portainer.
docker compose pull
docker compose run --rm guiltyspark codex login --device-auth
docker compose up

To build the container locally instead of pulling GHCR:

docker build -t guiltyspark:local .
GUILTYSPARK_IMAGE=guiltyspark:local docker compose up

For a local dry run:

uv sync --dev --no-editable
cp .env.local.example .env.local
uv run --no-editable --reinstall-package guiltyspark guiltyspark once

Run tests locally with:

uv run --no-editable pytest

Dashboard frontend

The dashboard UI is a React + TypeScript app in frontend/ (Vite). Its build output lands in src/guiltyspark/web/, which the Python backend serves via importlib.resources. That directory is git-ignored build output — the Docker image builds it in a Node stage, and the wheel force-includes it.

# Day-to-day UI work: Vite dev server (proxies /api to a running dashboard on :8343)
guiltyspark dashboard &            # or run the daemon/compose service
npm --prefix frontend install
npm --prefix frontend run dev      # open the printed http://localhost:5173

# Produce the bundle the Python server serves directly
npm --prefix frontend run build

Because src/guiltyspark/web/ is not committed, a bare guiltyspark dashboard from a fresh checkout serves nothing until you run the build once (or use the dev server). Docker builds handle this automatically.

Configuration

All settings are environment variables. The most important ones are:

Variable	Purpose
LOKI_URL	Base URL for Loki, for example http://loki:3100.
LOKI_QUERY	LogQL query to monitor, for example '{job=~".+"}'.
LOKI_LIMIT	Maximum log lines fetched per query range. Defaults to 5000.
LOKI_BEARER_TOKEN	Optional bearer token for an authenticated Loki instance.
LOKI_BASIC_AUTH	Optional user:password basic-auth credential for Loki.
CODEX_HOME	Persistent Codex auth/config directory. Defaults to /data/codex in Docker.
GUILTYSPARK_INTERVAL_SECONDS	Poll interval for daemon mode. Defaults to 300.
GUILTYSPARK_LOOKBACK_SECONDS	Initial lookback if no cursor exists. Defaults to 900.
GUILTYSPARK_MIN_EVENTS	Minimum events before a group of log lines counts as an incident. Defaults to 2.
GUILTYSPARK_MAX_INCIDENTS_PER_RUN	Cap on incidents analyzed in a single poll cycle. Defaults to 8.
GUILTYSPARK_MAX_REMEDIATIONS_PER_RUN	Cap on queued corrective measures attempted per target in one cycle. This throttles released observation backlogs and defaults to 3.
GUILTYSPARK_STATE_PATH	SQLite state path.
GUILTYSPARK_FINDINGS_PATH	JSONL findings output path.
GUILTYSPARK_RUNBOOK_PATH	Markdown runbook the agent reads before analysis.
GUILTYSPARK_NOTIFY_WEBHOOK_URL	Optional generic JSON webhook for new findings.
RESEND_API_KEY	Resend API key. Enables an email when guiltyspark opens a PR (only fires for its own PRs, never your manual ones).
GUILTYSPARK_NOTIFY_EMAIL_FROM	Verified Resend sender address for PR-opened emails.
GUILTYSPARK_NOTIFY_EMAIL_TO

Repository Targets

Fleet mode uses a TOML file containing one or more Loki-to-repository mappings. Start from targets.example.toml:

[[targets]]
id = "inventory-service"
loki_url = "http://loki:3100"
loki_query = '''{container=~"inventory-(api|worker)"}'''
github_repo = "example-org/inventory-service"
base_branch = "main"
mode = "observe"
test_commands = ["pytest -q"]
allowed_paths = ["src", "tests"]
max_changed_files = 8
expected_logs_path = "docs/EXPECTED_LOGS.md"

expected_logs_path (optional) is a repo-relative path to a document listing log lines the service emits that are expected or benign — deliberate warnings, startup chatter, retry notices. The Monitor fetches it from the target repository (private repos included, via the same GitHub auth) and hands it to Codex as context, so those patterns are not mistaken for anomalies. A missing or unreachable file is simply ignored — analysis proceeds without the extra context. The fetch is cached per GUILTYSPARK_EXPECTED_LOGS_CACHE_SECONDS.

Target modes are deliberately progressive:

• observe: detect and diagnose only; retain PR-recommended anomalies as held remediation dossiers. When promoting the protocol in the dashboard, the operator may explicitly release that backlog for corrective action.
• fix: clone, edit, enforce policy, and validate; never push.
• draft-pr: perform the same checks, then push a GuiltySpark branch and open a draft PR.
• pr: perform the same checks, then push a GuiltySpark branch and open a review-ready PR.

Production can supply the same structure as a JSON list through GUILTYSPARK_TARGETS_JSON; this is the preferred Portainer configuration path and takes precedence over the local TOML file.

Targets are stored in the SQLite state database and are editable from the dashboard (see below). GUILTYSPARK_TARGETS_JSON / GUILTYSPARK_TARGETS_PATH seed that store once, on first run while it holds no targets; after that the database is authoritative and dashboard edits persist across restarts. Changing the env var later has no effect unless the store is empty, and deleting a seeded target from the dashboard is not undone by a restart. The daemon re-reads targets from the store at the start of every poll cycle, so edits take effect within one interval without a restart.

For private repositories and PR modes, prefer a GitHub App installed only on the configured repositories. Set GITHUB_APP_ID, GITHUB_APP_INSTALLATION_ID, and either GITHUB_APP_PRIVATE_KEY or GITHUB_APP_PRIVATE_KEY_FILE. GuiltySpark mints and caches short-lived installation tokens for controller-owned Git and GitHub requests; Codex does not receive App credentials or installation tokens. A token provided through GUILTYSPARK_GITHUB_TOKEN_ENV remains available as a fallback when no App variables are configured. Partial App configuration is an error.

Onboarding A Repository

Adding a target to GuiltySpark does not automatically grant the GitHub App access to that repository. Complete all of these steps for every new repository:

1. Add the repository to the GuiltySpark GitHub App installation under GitHub Settings → Applications → Installed GitHub Apps → Configure → Repository access. The App needs Contents: read and write to push remediation branches and Pull requests: read and write to open PRs. This is required for public repositories too: public access permits cloning, but not pushing.
2. Add the repository target in the dashboard or seed configuration. Start it in observe mode and confirm that its Loki selector assigns the intended log streams.
3. Define narrow allowed_paths and representative test_commands, then promote the target progressively through fix, draft-pr, and pr as confidence grows.
4. Before relying on automatic remediation, replay a captured incident with --allow-push and confirm that the verification sequence passes and the App can open a PR.

If validation succeeds but publication ends with Permission to OWNER/REPO denied to APP-NAME[bot] and HTTP 403, first confirm that the repository is selected in the App installation. App credentials being present in the container is not sufficient; the installation token is scoped to its selected repositories.

Replaying A Captured Incident

Replay fixtures use the same generic incident and finding schema as the durable remediation queue. For local replay, copy the target example to an ignored config, point local_repo at the associated checkout, and use fix mode:

GUILTYSPARK_TARGETS_PATH=targets.local.toml guiltyspark replay \
  tests/fixtures/example-upstream-outage.json \
  --target inventory-service \
  --patch-output data/example.patch

Replay downgrades draft-pr and pr to fix unless --allow-push is supplied. This makes the same captured incident usable for local patch evaluation and an explicitly authorized draft-PR exercise.

Deployment

The Compose stack also serves the static marketing site through an nginx sidecar at http://localhost:8080. Set GUILTYSPARK_SITE_PORT to publish it on a different host port. The dashboard remains available separately on port 8343. Main-branch pushes publish site/ as ghcr.io/bheus/guiltyspark-site:latest before triggering the Portainer redeploy webhook, so the stack does not depend on repository bind mounts.

Push conventional commits to main. GitHub Actions tests the project, creates a semantic release, builds a native linux/arm64 image, publishes versioned and latest tags to GHCR, then triggers the GuiltySpark Portainer stack webhook so the stack re-pulls latest.

For the one-time Portainer setup, create the stack from this repository's main branch, enable the stack webhook in Portainer, and save its generated URL as the GitHub repository secret PORTAINER_GUILTYSPARK_WEBHOOK. The release workflow calls that webhook only after the new image has been pushed. If the secret is absent, the webhook step is skipped and Portainer's normal Git polling remains the fallback.

Supply target mappings and credentials through environment variables at deployment time. The repository intentionally contains no application-specific target profiles. Start each target in observe, promote it to fix after reviewing diagnosis quality, and use draft-pr or pr only after its validation commands and allowed paths are established.

Commands

guiltyspark once      # poll Loki once and analyze current incidents
guiltyspark daemon    # run forever
guiltyspark doctor    # validate configuration and connectivity basics
guiltyspark dashboard # serve the web dashboard (default port 8343)

Dashboard

guiltyspark dashboard serves a Monitor-voiced web console (default http://localhost:8343). In Docker it runs as the guiltyspark-dashboard compose service, which shares the daemon's /data volume and publishes port 8343. It shows catalogued findings, remediation history, and a live Loki view of recent error-severity events. Each live incident is classified into the target whose stream selector matches its labels; anything that matches no configured target is surfaced as an unassigned anomaly, so errors outside your containment protocols are still visible. A container picker (backed by GUILTYSPARK_DASHBOARD_FILTER_LABEL) narrows the survey server-side, so the LOKI_LIMIT budget is spent only on the containers you selected.

The dashboard is also the control surface for configuration:

• Containment protocols — add, amend, or decommission targets. Edits are validated with the same rules as the config file, written to the state store, and picked up by the daemon on its next cycle.
• Silence noise — an unassigned anomaly you judge to be noise can be silenced; it is suppressed from the stream (keyed by its incident fingerprint) and listed under Silenced anomalies. Silencing captures the anomaly's service, level, a sample line, and event count so the entry stays legible after it leaves the stream, and each entry carries an editable triage note for your own reference. Any entry can be restored.
• Silence patterns — a durable, service-scoped regex rule that suppresses current and future variants of a noise class, rather than one fingerprint at a time. With GUILTYSPARK_DASHBOARD_GROUPING enabled, Codex proposes the pattern and the UI shows its live blast radius; nothing is applied until you commit it. Rules carry an editable title and note, and an invalid regex is skipped rather than breaking the anomaly view.

With GUILTYSPARK_DASHBOARD_GROUPING enabled, clustering runs off the request path: a worker computes groups in the background, the API returns groups_pending while that is in flight, and the UI shows a cataloging state over the flat fallback list, polling every 2s until the groups land.

The page is a React + TypeScript app (in frontend/, see Development above) that talks only to the JSON API, so backend and client evolve independently. It polls every 60s and reconciles in place — open incident cards and in-progress edits survive a refresh. Read endpoints: /api/overview, /api/findings, /api/remediations, /api/anomalies?minutes=N[&containers=…], /api/containers?minutes=N, /api/targets, /api/anomalies/ignored (silenced fingerprints and pattern rules). Write endpoints: POST/DELETE /api/targets, POST/DELETE /api/anomalies/ignore, POST /api/anomalies/ignore-batch (silence a whole group), POST /api/anomalies/suggest-pattern (ask Codex for a pattern proposal), POST/DELETE /api/anomalies/rules, POST /api/anomalies/rules/metadata (edit a rule's title/note), and POST /api/anomalies/note (edit a silenced anomaly's triage note).

The dashboard is unauthenticated, and it can now modify configuration and trigger target changes. Keep it on a trusted LAN and do not expose it to the public internet.

Notes On Auth

The monitor uses Codex CLI instead of OPENAI_API_KEY. Sign in once with your ChatGPT account inside the container:

docker compose run --rm guiltyspark codex login --device-auth

The login is stored under CODEX_HOME, backed by the Compose-managed guiltyspark-data volume.