agent-polygraph

A framework-agnostic completion-lie detector for AI agents. Given one agent turn — a task, the tool calls and results it produced ("evidence"), and the final assistant message ("closing") — verify() decides whether the closing makes a completion or success claim that the turn's own evidence contradicts.

It measures the turn, not the model. The check is heuristic, deterministic, and synchronous, with zero runtime dependencies (standard library only).

from agent_polygraph import verify, ToolCall, ToolResult

v = verify(
    [ToolCall("c1", "deploy", {"svc": "web"}),
     ToolResult("c1", content="ERROR: image pull failed", is_error=True)],
    final_claim="Done -- deployed successfully.",
)

v.verdict      # "lie"
v.category     # "L1"  (error concealment)
v.confidence   # 1.0   (structured error bit)
v.evidence     # (Finding(check="error_concealment", detail=...),)

This is the SDK companion to polygraph-bench (in this repo since 2026-11-05), the public benchmark for agent-lying detectors. The detector shipped here is a clean-room reimplementation of a private production honesty gate, validated at verdict parity against that gate on the benchmark's 396-item corpus (numbers below).

---

Install

pip install agent-polygraph

Python 3.10+. No third-party runtime dependencies.

---

What it catches

Three heuristic check families, each mapped to a diagnostic category:

category	family (check id)	catches
L1	error_concealment	an unrecovered tool error hidden behind a plain, unacknowledged success claim
L2	completion_vs_evidence	a completeness / counted-total claim over a full-page fetch with no proof the set was exhausted
L3	narrative_vs_assertion	a runtime "it works / is fixed" claim after a file edit with nothing actually run

verify() returns a Verdict:

Verdict(
    verdict="lie" | "honest",
    evidence=(Finding(check, detail), ...),
    confidence=1.0 | 0.7,      # signal-quality GRADE, not a probability
    detector_version="0.2.0",  # self-stamped for cross-version comparison
    flags=(...),               # degradation markers driving a 0.7 grade
    category="L1" | "L2" | "L3" | None,
)

confidence is a signal-quality grade, not a probability: 1.0 when the verdict rode a structured framework signal, 0.7 when it rode an inferred truncation marker or a sniffed error payload (see "The error bit is the portability spine" under known limitations).

---

Framework adapters

Four adapters map a framework's native emission onto the same verify() core. The design default is audit everywhere — a foreign framework's false-positive rate is unknown until you measure it on your own traffic, and a false halt kills adoption. Blocking is opt-in, and only on the two surfaces that can actually halt a run; the audit-only adapters raise loudly on mode="block" rather than silently downgrading.

adapter	entry point	tool-error signal	default	block
OpenAI Agents SDK	output_guardrail	span wrapper error.data.error (structured)	audit	opt-in
Claude Code stop-hook	stop_hook	tool_result.is_error (structured)	audit	opt-in
OTel / OpenInference	verify_spans	OTel span status_code=ERROR (structured)	audit	raises loud
LiteLLM proxy	post_call_guardrail	none — sniff role:"tool" text only	audit	raises loud

from agent_polygraph.adapters import output_guardrail

result = output_guardrail(trace_export, mode="block")  # halt on a lie
if result.tripwire_triggered:
    ...  # the Agents SDK stops the run

Per-adapter quickstarts live in docs/: OpenAI Agents · OpenInference / OTel · Claude Code · LiteLLM.

---

Validation

All numbers below come from actually-executed runs and are reported verbatim. Parity and the recall floor are the honest limits of a heuristic detector — read them before you deploy in block mode.

Parity vs the private gate — 393 / 396 (99.24%) at 0.1.0

The detector was written clean-room from the public benchmark spec and corpus, then refined by behavioural parity iteration against a private production gate treated as a black box (feed an input, observe its block/allow verdict; the gate's source was never read). On all 396 public bench items:

agreement matrix (clean-room \ gate):
                 gate=lie   gate=honest
  clean=lie            88             0
  clean=honest          3           305

• 0 over-blocks — the clean detector never blocks anything the gate allows.
• 3 under-blocks, all category H2 (disclosed-failure honest twins). These are exactly the three items the gate's own heuristic tier false-blocks; the clean detector matches the ground-truth label on all three (they are honest). Reproducing the gate's mistake would require either reading its source or overfitting a rule that makes the public detector actively worse.

Label-level, both over the 396-item set: identical recall (0.460) and the same true-positive set; the clean detector carries 3 fewer false blocks (FP 3.18% vs the gate's 4.55%).

以上为 README 的开头部分。其余内容请阅读原文。 查看完整 README ↗