Sentinel
An agentic incident-response copilot that closes the loop: alert → investigation → human approval → exactly-once fix → verified recovery.
Website · Write-up: what a local 8B model taught us about letting AI touch production · User guide · Engineering handbook (PDF) · System design · ADRs · Benchmark report
Recorded from the running system: a bad deploy → the alert opens the incident → the agent proposes a cited rollback → approve → mitigated in 14 s.
🛡️ New in v2: Sentinel Guard, envelope protection for any AI agent
Sentinel's safety core now ships on its own. No agent changes your cluster unless policy, or a human, approved that exact change, and then only once. It works with coding agents running kubectl, CI and GitOps bots, and LangGraph or MCP agents, with no code changes, because it enforces at the Kubernetes API. Humans, CI and controllers are never sent to it, so if the guard is down they are unaffected and only agents wait.
helm install sentinel-guard oci://ghcr.io/abdulmdev/charts/sentinel-guard --version 2.0.0 -n sentinel-system --create-namespace
$ kubectl -n shop scale deploy/checkout --replicas=3 # as an agent
Error from server (Forbidden): ... Sentinel Guard: scale of shop/deployment/checkout needs human approval (request ar-c8f4166c-1).
$ kubectl sentinel approve ar-c8f4166c-1 # on-call engineer
Approved ar-c8f4166c-1: scale of shop/deployment/checkout (replicas: 2 → 3), by alice.
-
Bound to the exact change. Approving "scale to 3" doesn't admit "scale to 0", a second restart, or the same change after someone reverted it. This is proven on a real API server in CI, including the API server's own write retries (33 end-to-end checks).
-
Two people where it matters. approvals: 2 makes an action wait for two different people, each authenticated (by Kubernetes, or by OIDC at the guard), each approval signed. Approvers are notified when a request needs them.
-
Tamper-evident, and provable to others. Every request, decision and execution is a signed link in a hash chain; kubectl sentinel audit verify checks it yourself and also detects deleted records. kubectl sentinel audit export gives an auditor one file to verify offline (below).
-
Small. One Go binary, an 11.5 MB image, no database. Its state lives in the Kubernetes API, and 3 replicas share it.
-
Measured (p99 through the API server, 3 replicas, a laptop): a human's change 6.1 ms, an agent's pending change 7.5 ms, an approved change admitted once 34.6 ms.
-
Any agent, not just kubectl:
- Python and TypeScript SDKs:
guard.require(...), then guard.spend(...), or a decorator;
- a LangChain/LangGraph tool wrapper that tells the model to wait for approval;
- a zero-code MCP proxy that puts the guard in front of any MCP server's tools, for Claude Desktop or Code, Cursor and agent frameworks.
See the SDKs and the examples.
Also kubectl apply -f install.yaml, docker run ghcr.io/abdulmdev/sentinel-guard (single process, HTTP API for agents outside Kubernetes), and kubectl krew install sentinel/sentinel. Sentinel Guard docs →
Try it in one command, from a clone:
docker compose up
Then open http://localhost:8080 and sign in as Alice (token alice-demo; Bob's is bob-demo). The inbox opens on an Overview of what every agent did and what waits for people; click any request for its full, signed story. A support-desk agent works through tickets behind the guard: receipts go out on policy, every refund waits for you in the inbox (Approve once or Deny), closing an account needs both Alice and Bob, and erasure requests are never allowed. docker compose logs -f notifications shows what Slack would receive. Add --profile monitoring for Prometheus with the guard's drift alerts, and a Grafana dashboard. Quickstart →
Or open the repository in GitHub Codespaces (button above) and run scripts/guard-demo.sh, a scripted minute: a person approves, the action runs exactly once, a replay is refused, and the record is exported, verified offline and caught when edited. What it shows →
📊 SafeActionBench: how often do agents' actions cause harm?
An open, deterministic benchmark of 224 replayed production incidents: red herrings, missing data and prompt injection hidden in log lines.
- What it scores: the one action an agent proposes for each incident, as harmful, as a correct fix, or as an escalation to a human.
- Three levels: as proposed, after the system's own checks, and behind an independent reference guard that sees only metrics and deploy history.
- Any MCP agent can take it, with no code:
safeactionbench serve.
| dev split | Harmful, as proposed | Behind the reference guard | Fix rate |
|---|
| Always restart (the 3 a.m. reflex) | 83.0% | 0.0% | 20.6% |
| gemma4 8B, the model alone | 33.9% | 0.0% (38 blocked, no good fix lost) | 45.6% |
| Sentinel's rules | 0.0% | 0.0% | 82.6% |
The incidents are synthetic, so the guarded numbers are an upper bound; the benchmark says so on its own page. Leaderboard → · Take it →
🧾 Governance and compliance
What agents did, who allowed it and under which rules, in a form anyone can check:
- Evidence export.
kubectl sentinel audit export writes one file: every request, its signed events, every policy version and the signed chain head. kubectl sentinel audit verify --bundle checks it offline; --previous proves nothing was removed since the last archived export.
- Each decision names its rules. Requests carry the SHA-256 of the policy they were decided under; the guard keeps every version, signed.
- Guardian monitoring.
- Every replica re-verifies the whole chain every 5 minutes.
- Per-agent metrics and tested alert rules flag a broken or rewound chain, an approval backlog, and drift in an agent's behaviour: bursts of blocked attempts, rising denials, people overriding it.
- Mapped, not claimed: where this produces evidence for the EU AI Act (Articles 12, 14, 19, 26, 72; for Annex III high-risk systems they apply from 2 December 2027) and the NIST AI RMF, and what stays yours. docs/compliance →
✈️ Why Sentinel exists: the cockpit idea
Airliners don't rely on a pilot's nerve at 3 a.m. On-call engineers shouldn't have to either.
Modern airliners don't rely on a pilot's memory and nerve at 3 a.m. When something fails, Airbus's ECAM and Boeing's EICAS show the alert, the affected systems and the checklist to run. Airbus's flight-envelope protection stops inputs that would take the aircraft outside safe limits. The pilot stays in command throughout. The automation's job is to keep the right next step in front of them, so stress, fear or fatigue don't turn a manageable failure into a reactive one.
Production incidents have the same shape. An engineer gets paged at night, under pressure, with dashboards screaming, and the tempting move is the fast, reactive one: roll back whatever deployed last, restart everything, scale it up. Sentinel brings the cockpit discipline to on-call:
| In the cockpit | In Sentinel |
|---|
| Failure alert with the affected systems (ECAM / EICAS) | The alert opens an incident; the agent gathers deploys, logs, metrics and runbooks |
| The right checklist for this failure | One proposed action, with every claim cited to evidence |
| Envelope protection: unsafe inputs are blocked | Policy interlocks block unsafe actions even when proposed (no rollback without a recent deploy, no restart without memory growth, …) |
| The pilot decides | Nothing runs until a human approves |
| Exactly the commanded input, nothing more | A single-use capability executes exactly the approved action, once |
| Instruments confirm the recovery | Real-traffic probes verify the service is healthy before "resolved" |
| The flight data recorder | The audit trail, timeline and auto-written postmortem |
The goal isn't to replace the engineer. It's to make the calm, correct call the easy one, at any hour.
Sentinel is not affiliated with Airbus or Boeing; the comparison describes a design philosophy.
What it does
When production breaks, the on-call engineer's first 10–30 minutes go to gathering evidence. Sentinel does that work, keeps a human in charge, and proves the result:
- Detects: Prometheus fires; the incident opens itself (deduplicated, and correlated when several rules trip on one fault).
- Investigates: a LangGraph agent calls MCP tools (deploys, grouped logs, metrics, runbooks) within hard step and cost budgets.
- Diagnoses with evidence: ranked root causes, each cited to evidence gathered in that run; deterministic guardrails block unsafe proposals.
- Waits for a human: nothing touches production before approval.
- Executes exactly once: approval mints a 5-minute, single-use capability for that one action on that one service.
- Verifies and measures: real-traffic probes decide
RESOLVED, time to mitigate is recorded, and the postmortem writes itself.
Results (all measured)
| |
|---|
| Time to mitigate (alert start → verified healthy, incl. the human approval) | 14–41 s across 6 live runs (phase history) |
| Rule baseline, 112 replayed incidents (incl. red herrings, outages, prompt injection) | 92.9% correct remediation, 0 harmful actions, 0 successful injections, 0 PII leaks |
| Local LLM (gemma4 8B), same 112 incidents | it proposes a harmful fix in 33.9% of incidents; the guardrails let 0% through (62.5% correct; every unsafe proposal escalates instead) |
| The eval caught a real bug | the first baseline rolled back harmless deploys in 16% of incidents → fixed to 0%; CI now fails on regression |
| Load (k6) | incidents p95 314 ms at 5/s; search p95 18 ms at 20/s; a 200-request burst admits exactly 120 (distributed limiter) |
| Quality gates | 331 automated tests (incl. 72 for Sentinel Guard, 25 for its SDKs, 13 for SafeActionBench), plus 33 Guard checks on a real API server and an 8-check upgrade test from the last alpha, promtool-tested alert rules, an asserted walkthrough and a checked docker compose quickstart; CI with a real Kubernetes end-to-end run, green on the first push (only the paid Claude eval job is skipped: no key) |
Why it's interesting to build
🤖 Agentic AI, done safely
- LangGraph state machine with human-in-the-loop interrupt and Postgres checkpoints (survives restarts, resumes on any replica)
- Tools over MCP; budgets enforced by the graph (≤ 8 tool calls, ≤ $0.50)
- "The model proposes, policy disposes": PII redaction, injection removal, citation validation, runbook interlocks, constrained decoding
- Pluggable LLM: Claude, or local gemma4 via Ollama, or none (rules)
🔐 Zero-trust security
- Only the gateway mints service credentials and only Sentinel Guard issues capabilities: ES256 + JWKS; every other service verifies and can't forge
- Single-use capabilities bound to the exact action and parameters, redeemed once at the guard, which fails closed
- Users via OIDC (Keycloak, PKCE); every hop authenticated and tenant-bound
- Row-level security everywhere, including the agent's checkpoints; per-pod least-privilege secrets checked in CI
⚙️ Distributed systems
- Kafka ingestion with idempotent re-index and a DLT; hybrid retrieval (pgvector + full-text, RRF) in one Postgres transaction
- Race-safe alert dedup via a per-service advisory lock (8 concurrent deliveries → 1 incident, tested)
- Atomic Lua token bucket on Redis, with a local fallback
- Semantic cache invalidated by a knowledge-version fingerprint
📈 Engineering rigor
- Offline, deterministic benchmark (112 incidents + held-out set) as a CI gate
- One OpenTelemetry trace per incident across Java → Python → MCP; Grafana for cost, safety and time to mitigate
- Helm (hardened pods, NetworkPolicies), kind e2e in CI, Terraform for EKS/RDS/MSK
- 9 ADRs and a 16-chapter handbook with every bug found and how it was fixed
Architecture
| Service | Stack | Role |
|---|
gateway-service | Java 21, Spring Boot 4, Spring AI 2 | API, OIDC + tenancy, rate limits, hybrid RAG /ask, alert intake, sole token issuer (JWKS) |
agent-service | Python, LangGraph, FastAPI | Investigation → RCA → guardrails → approval → execute → verify → postmortem |
ingestion-service | Spring Boot, Kafka, Flyway | Chunk → embed (local ONNX) → index; owns the schema (V1–V7) |
mcp-tools/observability | MCP (FastMCP) | Deploys, grouped logs, metrics, verification probes |
mcp-tools/remediation | MCP (FastMCP) | Capability-gated rollback / restart / pool / failover / scale |
web | Next.js 16 BFF | Console: OIDC code flow, approvals, closed-loop timeline, integrations |
chaos-lab | FastAPI | Demo services with injectable faults and an ops API |
Deep dive: high-level design · security · evaluation · the full handbook as a PDF.
Screenshots
| Closed loop: executed, verified, timeline | One trace per decision (Jaeger) |
|---|
| |
| Grafana: outcomes, time to mitigate, LLM cost | Revocable integration tokens (tenant admins) |
| |
Run it
Prerequisites: Docker, Java 21+, Maven, uv, Node 24. Optional: Ollama for a free local LLM.
scripts/dev-local.sh up
python3 chaos-lab/inject.py checkout bad_deploy
Open http://localhost:3000, sign in with Keycloak (alice@acme; dev passwords in deploy/keycloak/sentinel-realm.json), wait about 40 s for the alert-opened incident, and click Approve. With a local model: SENTINEL_LLM_PROVIDER=ollama scripts/dev-local.sh up. On Kubernetes: scripts/kind-up.sh. Full walkthrough: user guide.
Tests and quality gates
cd gateway-service && mvn verify
cd agent-service && uv run pytest
uv run --project agent-service python evals/rca/run.py --check
cd guard && go test -race ./...
LOAD=1 scripts/guard-e2e.sh
CI (workflow) runs all test suites (Testcontainers on real pgvector/Redis), the benchmark gate (main + held-out), docs-sync checks, Helm/kubeconform/Terraform validation, a least-privilege secrets check, and a real kind cluster end to end (incident → approve → RESOLVED, NetworkPolicy denials), and the Sentinel Guard end to end (33 checks, a load run, the guard taken down), and the docker compose quickstart (29 checks, with Prometheus and Grafana), then publishes multi-arch images.
Where to look in the code (5-minute tour)
How it was built
Nine phases, each with its goal, decisions, bugs found and fixed, and measured results in the phase history:
| Phase | Outcome |
|---|
| 1 Foundation | Gateway ↔ LangGraph agent ↔ chaos-lab; human approval interrupt; tenancy from the JWT |
| 2 Ingestion + RAG | Kafka ingestion; hybrid pgvector + full-text retrieval; RLS enforced by a non-superuser role |
| 3 Agents | MCP tool servers, downscoped tool tokens, budgets, durable Postgres checkpoints |
| 4 Evaluation | 112 replayed incidents, deterministic scoring, CI gate; found the 16% harmful-rollback bug |
| 5 Hardening | Guardrails + injection tier, Redis limits, semantic cache, OpenTelemetry, k6 |
| 6 Ship | Next.js console, Helm on Kubernetes, Terraform for AWS, CI |
| 7 Close the loop | Alerts open incidents; capability-gated execution; verification; time to mitigate |
| 8 Zero trust | ES256 + JWKS, Keycloak OIDC, every hop authenticated, checkpoint RLS |
| 9 Publish and measure | Full local-LLM benchmark; runbook preconditions take gemma4 from 11.6% to 0% harmful actions |
| 10 Sentinel Guard (v2 alpha) | The safety core as a Kubernetes-native Go service for any agent; exactly-once proven on a real API server, including its write retries; p99 42.9 ms for an approved change |
| 11 The copilot on the guard | One safety core: the agent proposes to the guard, console approvals are recorded there by the gateway, the adapter redeems the guard's capability; live: alert → verified fix in 24.4 s |
| 12 SDKs and the MCP proxy | Python and TypeScript SDKs, a LangChain/LangGraph wrapper and a zero-code MCP proxy, tested against the real guard |
| 13 SafeActionBench | The incident benchmark as an open benchmark for any agent and any guard; an independent guard alone took a small model from 33.9% to 0% harmful |
| 14 Governance and compliance | A signed evidence export anyone can verify offline, policy versions, signed checkpoints, guardian alerts on agent drift (tested with promtool), the EU AI Act / NIST AI RMF mapping, and a one-click Codespaces demo |
| 15 Try it, ship it | docker compose up with a live agent, an approvals inbox in the guard, the guard's Grafana dashboard, signed charts and SDK packages on every release, with PyPI/npm/Artifact Hub publishing ready to switch on |
| 16 Two people | approvals: N for the riskiest actions (distinct, authenticated people, every approval signed and verified), approvers verified by OIDC at the guard, notifications to Slack-style webhooks; v2.0.0 released |
| Decision | Reason |
|---|
| Java gateway + Python agents | Platform concerns (auth, tenancy, rate limits, resilience) sit on the JVM. The agent ecosystem (LangGraph) is in Python. See ADR-0001. |
| Human-in-the-loop interrupt | Remediation changes production. LangGraph interrupt() pauses the graph, and the gateway takes the reviewer's identity from the JWT, not the request body. |
| Model routing | Triage is classification, so it runs on claude-haiku-4-5. RCA needs reasoning, so it runs on claude-opus-5-5. |
| Heuristic fallback | Without an API key, or when an LLM call fails, rule-based triage and RCA run instead. They are also the eval baseline the LLM path has to beat. |
| Tenant from the token, everywhere | The gateway ignores tenant_id in request bodies. Agent threads are keyed tenant:incident. Postgres uses row-level security. |
| chaos-lab | Real services with injectable faults, so the agent is tested on real symptoms instead of mocked text. |
| Hybrid retrieval in Postgres | Vectors and full-text live in the same row and are fused with RRF. One system, one transaction, one RLS model. See ADR-0003. |
| Tools over MCP | Observability and runbook search are MCP servers the agent discovers at runtime, reusable by any MCP client. See ADR-0004. |
| Downscoped tool tokens | The agent never holds a user's token: the gateway mints a 15-minute, single-tenant, knowledge:read, MCP-only JWT per incident, and it's never checkpointed. |
| Capabilities, not deploy credentials | The agent holds no standing power to change production. It registers each proposal in Sentinel Guard; when a human approves in the console, the gateway records that decision in the guard on their behalf, and only then does the guard issue a single-use capability bound to one action, one service, the approved parameters and one incident. The adapter verifies it with the guard's public key and redeems it at the guard before acting, failing closed. A compromised agent can at most do the one thing a human approved, once. See and . |
Roadmap
Claude benchmark column (needs an API key) · compaction after archived evidence export · approvers from different teams · Slack approvals · KMS-backed signing and mTLS · async approval with streamed progress · dependency-aware alert correlation · incidents recorded from real postmortems. Details: limitations and roadmap.
Contributing and license
Contributions are welcome: see CONTRIBUTING (how to run, verify and keep the docs in sync), the code of conduct and the security policy. Licensed under the Apache License 2.0.
