Project Beacon
Try an agent on realistic work before trusting it with real work.
beaconlab.dev replays recorded runs in your
browser — the same evidence bundles this repository writes, stepped through
check by check, with nothing to install. It is the fastest way to see what a
verdict is made of before deciding whether to clone anything.
Point Beacon at an agent, give it a scenario built on synthetic mail and files,
and get back an evidence bundle that says what the agent actually did:
$ python3 -m beacon run inbox-briefing
PASS: Inbox briefing with draft-only replies
Evidence: .beacon/runs/run-29c153bbe9f5/evidence.json
Report: .beacon/runs/run-29c153bbe9f5/report.md
Nothing here is a language-model judging another language model. Every
assertion is a deterministic check against recorded tool calls and the
before/after state of a simulated service.
The interesting run is the one that fails. Below, the same scenario twice — an
agent that does the work, then one that does the work and tries to send the
drafts. Scenario policy blocks the send either way, so both mailboxes end
identical and after.mail.sent == [] is true for both. What separates them is
that Beacon recorded the attempt:
Recorded from a real run by tools/demo.tape, which is
committed, so the demo cannot drift from what the tool actually prints.
Contents
Why this exists
Deciding whether to trust an agent usually means one of two bad options: read
its prompt and guess, or connect it to a real inbox and find out. The first
proves nothing. The second is the experiment you cannot undo.
Beacon is the third option. A scenario seeds synthetic services, scopes a tool
surface, states a goal, and lists assertions that can fail. The agent runs
against that instead of your data. Beacon records every tool call, snapshots
state before and after, evaluates the assertions, resets the services, and
verifies the reset — then writes it all down.
Two properties do most of the work:
"Not run" never becomes a pass. A subject that crashed, timed out, or
produced nothing measurable resolves to INCOMPLETE, not FAIL and never
PASS. Silence is not evidence of good behaviour.
Assertions have to be falsifiable. An assertion nobody has watched fail is
a claim the evidence does not support. tests/test_falsifiability.py fails the
build if a behavioural assertion exists that no subject can break — a check
that found two assertions already shipped here that could not fail, both of
which report.md had been printing as findings.
How this differs from what you may already run
Not another agent framework, and not a replacement for your tracing.
| What it does | What it does not |
|---|
| Agent runtimes — LangGraph, n8n, MCP hosts, your own loop | Own the tools and the control flow | Say whether what happened should have |
| Tracing and evals — spans, cost, latency, LLM-as-judge | Answer how long, how much, and roughly how good | Grade against a check declared before the run, or hold still between runs |
| Beacon | Grades against assertions declared in advance, records the attempt and not only the outcome, says INCOMPLETE when it could not tell, and hands you the bundle it decided from | Execute your agent, or judge anything with a model |
Beacon sits underneath whatever you already use: your agent brings its own
model and its own loop, and Beacon supplies the world it runs in and the
verdict about what it did there. A trace tells you an agent called
mail_send_draft and it took 240ms. Beacon tells you the call was refused,
that the refusal is why the mailbox looks untouched, and that the assertion
about it failed — which is the difference between a log and a result.
Quickstart
pip install project-beacon
The core has no runtime dependencies. Or clone it and run python3 -m beacon
straight out of the checkout — the scenarios and the taxonomy ship inside the
package either way:
여기까지가 README 앞부분입니다. 나머지는 원문에서 읽을 수 있습니다. README 전문 보기 ↗
