SRE Lab
A reliability-first lab, live at lab.sallyroth.dev: small, working SRE tools (an SLO burn simulator, Down or Me uptime triage, The Incident Files postmortem search, agent evals) and illustrated, step-through explainers on incident management, observability, Kubernetes, Linux, infrastructure as code, delivery and security. Engineering with AI agents is its own section.
Tools
| Tool | What it does |
|---|
| SLO Burn Simulator | Pick an SLO and a burn rate; see how long the error budget lasts, which multi-window burn-rate alerts fire (Google SRE Workbook thresholds), and copy matching Prometheus recording and alerting rules (one page alert and one ticket alert per service). Static page, no backend. |
| Down or Me | Uptime checks every 5 minutes on a Cloudflare Worker. When something looks unusual, Jev classifies the cause (real outage, degraded, monitor-side, planned maintenance, blip) with a confidence score, and an email goes out only for confident, real problems. Uncertain calls are left for a human. |
| Did the Agent Do It Right? | Three simulated agent runs on the same synthetic incident (one correct, one that checks the wrong service but sounds sure, one stuck in a retry loop), judged by their tool-call traces with span-based Pydantic Evals checks, not by how convincing the final explanation sounds. Static page, no LLM, no API calls. |
| The Incident Files | 318 real public postmortems, searchable by failure mode (an in-browser embedding model plus BM25, merged with reciprocal rank fusion), and nine recurring failure patterns with how companies prevented them. Static page, no backend, no API calls. |
Explainers
Step-through diagrams rendered from chapter specs in <topic>/parts/*.js by the shared engine in drawn/. Each chapter leads with the system across many teams, then a short summary, gotchas, questions to think through, and links to the primary sources it is based on. <topic>/check.mjs checks each one (spec integrity, layout, source links); the spec format is in k8s/README.md.
| Page | Covers |
|---|
| Kubernetes | 18 chapters: control loop, kubectl apply, scheduling, Services, DNS, storage, autoscaling, RBAC, upgrades, a troubleshooting map |
| Linux | Page cache, the OOM killer, cgroups, a packet's path, conntrack, TCP queues, the first 60 seconds of performance analysis |
| Incident Management | The incident program as a system: severity contract, routing and ownership, concurrent incidents, comms and legal clocks, fleet on-call health, learning, follow-through, readiness, metrics |
| Taming Cardinality | How labels multiply series and cost, finding the culprit, dropping, aggregating and governing |
| Terraform, Five Years On | moved/import/removed blocks, checks and tests, OpenTofu, ephemeral values, splitting state |
| Infrastructure as Code | Remote state and locking, modules, environments, plan/apply in CI with policy checks, drift, testing |
| Delivery | CI as a shared service, build once and promote, GitOps with Argo CD, canaries, feature flags, schema migrations, supply-chain signing, merge queues |
| Observability | OpenTelemetry and the Collector, Prometheus scraping and PromQL, tracing and sampling, Loki, alerting |
| Security | Org guardrails and policy as code, federated identity, secrets and supply chain programs, vulnerability flow, fleet detection, agents, program metrics |
AI engineering
The full dated changelog of research passes is at changelog/. The landing page (index.html) shows only the latest three entries: add each new entry at the top of both, and drop the oldest from the landing page. The landing page groups tiles as Start here, Tools, Reliability explainers, and AI engineering. scripts/stamp-updated.py restamps each page's "updated" date from git history.
Checks
for d in */check.mjs; do (cd "$(dirname "$d")" && node check.mjs) || exit 1; done # explainer specs
node worker/test-down-or-me.mjs # Down or Me cron cycle: triage retries, notification (Node 22+)
node worker/test-access.mjs # private board: Access JWT checks, fails closed
(cd incidents && npm ci && node evalnode.mjs .) # search quality
How Down or Me works
cron (5 min) ─► HTTP / status-page / game-server checks ─► D1 history
│
unusual? ── no ─► done (no AI call)
│ yes, first time or every 30 min while it lasts
daily Jev budget left? ── no ─► show on board only
│ yes
Jev: cause + confidence + "notify?" ─► D1 verdict
│
confident, real, not already sent ─► email
- Jev is called only from the Worker. The API key is a Wrangler secret and never reaches the browser.
- A daily cap (
JEV_DAILY_CAP, default 200) bounds cost even if something loops.
- The public board lists only generic services. The private board requires Cloudflare Access:
/private/api/status verifies the Access JWT (signature, issuer, audience, expiry, owner email) and fails closed (403) unless ACCESS_TEAM_DOMAIN, ACCESS_AUD and OWNER_EMAIL are set. The cf-access-authenticated-user-email header alone is never trusted.
Layout
index.html lab landing page: tools, explainers, latest changelog entries (Cloudflare Pages)
changelog/index.html full changelog of daily research passes
slo/index.html SLO Burn Simulator
status/index.html Down or Me board, public (/status/)
private/status/ Down or Me board, owner only (/private/status/)
agent-evals/ Did the Agent Do It Right? (page, data/, build/ for the Python eval run)
incidents/ The Incident Files (page, data/, build and eval scripts)
functions/ Pages Functions: /api/status, /private/api/status
worker/ Cloudflare Worker: cron checks, Jev triage, migrations, offline tests
drawn/ shared diagram engine and styles for the explainers
<topic>/ one explainer each: index.html, parts/*.js, check.mjs
fleet/, ownership/ single-page explainers
scripts/ maintenance (stamp-updated.py)
Hosting
- Pages project
sre-lab serves the pages and the status API (Pages Functions in functions/, D1 binding). Custom domain lab.sallyroth.dev via a CNAME at the DNS host.
- Worker
down-or-me runs only the 5-minute cron (no public URL) and writes to the same D1 database.
- Alerts are sent with Resend. The private board is at
sre-lab-97p.pages.dev/private/status/ behind Cloudflare Access (an Access application on both hostnames for /private/*; set ACCESS_TEAM_DOMAIN = <team>.cloudflareaccess.com and ACCESS_AUD = the application AUD tag as Pages env vars).
Setup (owner)
npx wrangler d1 create down-or-me # paste the id into both wrangler.toml files
cd worker
npx wrangler d1 migrations apply down-or-me --remote
npx wrangler secret put JEV_API_KEY
npx wrangler secret put RESEND_API_KEY
npx wrangler secret put NOTIFY_TO
npx wrangler secret put PRIVATE_TARGETS
npx wrangler deploy
cd ..
npx wrangler pages deploy . # then set PRIVATE_TARGETS and OWNER_EMAIL on the Pages project
How The Incident Files works
build time (incidents/build.mjs, Node) browser (incidents/index.html)
incidents.json ─► all-MiniLM-L6-v2 (q8) ─► query ─► same model ─► 384-dim vector ─► dot product ─► top 5
embeddings.bin (384-dim) query ─► BM25 over the same text ───────────────────────► top 5
- Corpus: real public postmortems, found through danluu/post-mortems. Each entry holds facts (company, date, trigger, impact, detection, fix) and a short summary written for this demo, plus a link to the original. No text is copied from the sources.
- Same model at build time and in the browser, pinned to Transformers.js 3.8.1, so query and document vectors are comparable.
- Ranking: the embedding model matches by meaning, BM25 catches exact terms; reciprocal rank fusion merges them, with keyword ranks at half weight so one shared word can't outvote meaning.
incidents/data/eval.json and node evalnode.mjs . score it offline.
- Patterns (
incidents/data/patterns.json): recurring failure modes proposed by the embedding model and confirmed by hand; each prevention cites the incidents whose own follow-ups it comes from.
- Rebuild after editing the corpus:
cd incidents && npm install && node build.mjs --check, then node evalnode.mjs . to score AI, keyword and hybrid search offline.
How Did the Agent Do It Right? works
build time (agent-evals/build/run_evals.py, Python) browser (agent-evals/index.html)
scripted agents ─► mock tools ─► OTel spans ─► pydantic-evals Dataset ─► data/results.json ─► runs, traces, checks
(execute_tool *) evaluate_sync + evaluators data/report.txt (plain-text report)
- Simulated, synthetic: three made-up incidents × three deterministic Python "agents" (correct, wrong service, retry loop). No LLM, no keys, no API cost; the page is static at runtime and only reads
data/results.json.
- Checks read the trace:
HasMatchingSpan per case (metrics, logs, deploys checked on the affected service), the built-in MaxToolCalls, and custom span-tree evaluators for a retry budget and "no rollback without evidence". Output checks (names the service and cause) count too; a "sounds confident" label is reported but never scored.
- Offline build: Logfire is configured with
send_to_logfire=False, spans stay in memory, and a socket guard fails loudly on any outbound connection. Fake durations keep the output stable; only generated_at changes between runs.
- Rebuild (venv lives outside the repo; exact versions in
agent-evals/build/requirements.txt):
python3 -m venv ~/.venvs/agent-evals && ~/.venvs/agent-evals/bin/pip install -r agent-evals/build/requirements.txt
# If venv fails because ensurepip is missing (no python3.12-venv package), create it without pip and bootstrap pip:
# python3 -m venv --without-pip ~/.venvs/agent-evals && curl -sS https://bootstrap.pypa.io/get-pip.py | ~/.venvs/agent-evals/bin/python
~/.venvs/agent-evals/bin/python agent-evals/build/run_evals.py
