Self-diagnosing server

When a service on my Raspberry Pi 5 home server goes down, this agent collects evidence, has an AI explain what happened, writes an incident page and sends one phone notification with the likely cause and the commands to fix it. It never runs anything itself: it only diagnoses, and a human decides.

It has run on the Pi since October 2026, watching about 30 services (Docker containers, systemd services, DNS, backups, disks, a Cloudflare tunnel).

Why it is built this way

• Privilege separation. Only the collector runs as root, and it can only run commands that are written in the code. The config file picks from built-in checks (http, dig, cert, backup_status) and lists container/unit names, which are checked against a strict pattern. A config file can never add a command.
• The AI is untrusted. It runs as a separate unprivileged user with every tool disabled, gets only redacted text, and must answer in a fixed JSON schema. The prompt tells it the evidence is untrusted data (log lines could contain prompt injection). Its answer is validated, and suggested commands that match a blocklist (rm -r, prune, fsck, mkfs, dd, umount, reading .env files, ...) are dropped before anyone sees them.
• Nothing is executed from the answer. The page says "run them yourself".
• Redaction before anything leaves the machine: bearer tokens, key=value secrets, notification topic URLs, URL query strings, e-mail addresses, long hex strings and base64-like blobs.
• Cost control. Gemini (through the Gemini CLI) is tried first and is free at this volume. Claude (Claude Code in print mode) is only the fallback, capped per run (--max-budget-usd) and optionally per month through a shared budget module. Plus: one diagnosis per monitor per 30 minutes, 10 per day.
• Sandboxed. The systemd unit uses ProtectSystem=strict, so the whole filesystem is read-only for the agent except its state and the incident pages.

Lessons the code encodes

Real false alarms that changed the code:

• "The HDD is mounted read-only": wrong, twice. Inside the ProtectSystem=strict sandbox every path looks read-only. The collector now reads the host's mount table (findmnt --task 1, marked "host view"), and the prompt only allows calling a disk read-only if those options start with ro,.
• "Unmount and remount the disk": a dangerous suggestion on a server with Docker bind mounts. Banned in the prompt and filtered by the blocklist.
• Flapping monitors caused useless diagnoses, so a DOWN is re-checked after 120 seconds first.
• known-issues.txt gives the model the server's history (e.g. "the HDD used to drop on the old power supply", "get_throttled=0x80000 is a past soft temperature limit, not a power problem"), which made the diagnoses noticeably more specific.

Also in this repo: jobwatch

jobwatch/ closes a gap Uptime Kuma has: is every scheduled job still running?

• jobrun NAME MAXAGE command... wraps each cron job and records its last success in ~/.jobs/.
• jobwatch.py (cron, every 5 min) pushes up/down to a Kuma push monitor "Scheduled jobs", naming every job whose last success is older than its MAXAGE and every watched systemd timer whose last run failed for 20+ minutes. If jobwatch itself stops, the push monitor goes down on its own.
• So a silently broken cron job becomes a normal Kuma outage, which diag-agent then diagnoses.

*/30 * * * * jobrun my-sync 7200 /usr/bin/python3 /opt/my-sync/sync.py
*/5  * * * * /usr/bin/python3 /usr/local/bin/jobwatch.py

When you remove a cron job, also delete its ~/.jobs/<name>.ok file, or jobwatch keeps reporting it as late.

Setup

Requirements: Python 3.11+ (standard library only), systemd, Docker (for container checks), Uptime Kuma with a status page that lists the monitors (the agent reads its public JSON API), and at least one of:

• gemini-ask: any command that takes the system prompt as its last argument and the input on stdin, accepts --tag, --timeout, --effort and --schema, and prints the JSON answer (mine runs the Gemini CLI as a locked-down user), or
• Claude Code installed for an unprivileged user (claude -p works for that user).

sudo install -m 755 diag-agent.py /usr/local/bin/diag-agent.py
sudo mkdir -p /etc/diag /var/lib/diag /var/www/incidents
sudo cp examples/monitors.json examples/known-issues.txt /etc/diag/     # then edit both
sudo cp systemd/diag-watch.* /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now diag-watch.timer

sudo diag-agent.py list                  # monitors from the config
sudo diag-agent.py collect "Paperless"   # print the redacted evidence, no AI
sudo diag-agent.py test "Paperless"      # evidence + AI diagnosis, printed, no notification

Tests

python -m unittest discover -s tests -v   # standard library only; runs in CI on Python 3.11 – 3.13

They cover the parts that keep the agent safe: redaction of secrets, the JSON-schema validation and the command blocklist, config values that try to inject commands, HTML escaping and pruning of incident pages, and jobwatch.

Monitor names in monitors.json must match the monitor names in Kuma. A monitor that is not in the config still gets the generic evidence.

Settings (environment variables)

Variable	Default
KUMA_URL	http://127.0.0.1:3001	Uptime Kuma
KUMA_STATUS_PAGE	home	slug of the status page with the monitors
DIAG_CONFIG / DIAG_KNOWN	/etc/diag/monitors.json / known-issues.txt
DIAG_SITE	/var/www/incidents	where incident pages are written (kept 90 days)
DIAG_PUBLIC_URL	http://localhost/incidents	link in the notification
DIAG_NOTIFY	/usr/local/bin/notify	notify <channel> <prio> <title> <msg> <tags> <url>, e.g. an ntfy wrapper
DIAG_ENGINE / DIAG_FALLBACK	gemini / claude	claude / none
DIAG_GEMINI_ASK	/usr/local/bin/gemini-ask
DIAG_CLAUDE_USER / DIAG_CLAUDE_BIN	claude-runner / ~/.local/bin/claude
DIAG_MODEL	sonnet	Claude model for the fallback
DIAG_MONTHLY_USD	3	monthly cap if ai_budget.py is present in /usr/local/lib
DIAG_BACKUP_STATUS	/var/backups/STATUS	file read by the backup_status check

License

MIT