Can your AI judge decide alone?
judge-audit replays decisions your people already made and tells you how much of that work an AI judge can take over, at the error you accept, with the bound attached. Open source, runs in CI, no LLM inside the measurement, and every published number recomputes from committed evidence.
What we contribute.
- An independent audit, not a model. judge-audit measures the confidence an AI judge declares against decisions people already made. It judges nothing itself.
- Evidence anyone can recompute. Every published figure regenerates from committed checkpoints, and CI fails when a report or this README drifts from its evidence (
scripts/verify_published.py, scripts/verify_readme.py).
- Pre-registered studies, separate reviews. The v0.5 and v0.6 studies were pre-registered before their first model call (v0.6: the plan's sha256 committed at the freeze; amendments A1–A4 hashed before the calls or the analysis they govern, A5, an erratum on wording, at publication). Each pull request goes through a separate review and QA pass before it merges, by AI review agents that did not write the change, with the verdicts posted on each pull request. No external human reviewer has read the study plans.
- Our own security advisories. We publish the weaknesses we find in our own tools: GHSA-52xj-j738-jpm3 (judge-audit: an API key in error output, or sent on a redirect) and GHSA-f7jx-f5h6-pmx7 (agent-assurance).
| |
|---|
| 🎯 Safe automation rate | The share of decisions a judge can take alone at the error you accept (1, 2, 5, 10 %), the confidence threshold to deploy, and an out-of-sample check. |
| 🌡️ Calibration | When it says 90 %, is it right 90 % of the time? ECE, Brier score, reliability diagram, with intervals. |
| 🚦 CI gate | judge-audit check fails the build when the judge drifts or the rate falls below your minimum. |
| 🤖 Where you work | CLI, GitHub Action with a PR comment, and an MCP server for Claude Code, Cursor or any agent. |
Quickstart · What you get · v0.6 findings · v0.5 findings · Judges supported · ▶ launch video
v0.6 findings
v0.6 in short. 12 AI judges, the same 2,974 items of MASSIVE 1.1 in English, Spanish and Catalan, pre-registered before the first model call (plan, sha256 committed at the freeze; amendments A1–A5). Six pre-registered transfer tests asked whether a threshold validated on English rows at a 5 % error target lets more than 5 % errors through on the same items in Spanish or Catalan: none was resolved at its Holm-corrected level, so no breach was shown; that is not a pass: the approximate one-sided 95 % upper bounds on the pooled target-language error run from 3.30 % (T6) to 8.65 % (T2), above 5 % in 5 of 6 tests, and 3 of the 6 came out as predicted. Demonstrated on this sample, and only on it (MASSIVE is translated public data, probably seen in pretraining): declared confidence does not always match accuracy; mean confidence minus accuracy is +0.080 to +0.085 for jev, −0.052 to −0.067 for gpt-6-luna and +0.063 to +0.099 for claude-sonnet-4.5 across the three languages. The differences between languages are 95 % intervals with no correction for multiplicity, so they are not called demonstrated (§2). The number to act on is the same audit on your own decisions.
At a glance: v0.6, MASSIVE 1.1, 2,974 items in English, Spanish and Catalan, 12 judges: the held-out share automated by thresholds chosen at a 5 % error target (the threshold chosen on one half of the items, applied to the other); met when its bound is within 5 %.
| Judge | en-US | es-ES | ca-ES |
|---|
| jev | 29.0 % (bound 5.8 %, not met; splits 30.7–55.6 %) | 25.3 % (bound 7.4 %, not met; splits 0.0–23.6 %) | none (splits 0.0–19.5 %) |
| gpt-6-luna | 18.2 % (bound 4.6 %, met; splits 21.8–39.4 %) | none (splits 0.0–13.3 %) | 5.3 % (bound 9.8 %, not met; splits 0.0–0.0 %) |
| claude-sonnet-4.5 | 21.5 % (bound 2.9 %, met; splits 21.5–21.5 %) | 8.5 % (bound 5.7 %, not met; splits 0.0–8.8 %) | none (splits 0.0–6.9 %) |
| clef-flash (exploratory) | 40.6 % (bound 5.0 %, met; splits 0.0–44.4 %) | 38.5 % (bound 5.7 %, not met; splits 4.0–44.2 %) | 39.0 % (bound 4.2 %, met; splits 18.0–60.0 %) |
| decider-2b (exploratory) | 31.1 % (bound 7.7 %, not met; splits 0.0–9.7 %) | 10.4 % (bound 7.4 %, not met; splits 0.0–4.5 %) | none (splits 0.0–0.0 %) |
| decision2-kai-0.6b (exploratory) | 4.1 % (bound 7.3 %, not met; splits 0.0–7.4 %) | none (splits 0.0–0.0 %) | none (splits 0.0–0.0 %) |
| deepseek-r1 (exploratory) | none (splits 0.0–28.7 %) | none (splits 0.0–0.0 %) | none (splits 0.0–0.0 %) |
| gemini-3.6-flash (exploratory) | 55.1 % (bound 3.4 %, met; splits 55.1–55.1 %) | 26.6 % (bound 5.3 %, not met; splits 26.0–52.8 %) | 27.1 % (bound 5.1 %, not met; splits 44.6–53.6 %) |
| laya-multilingual (exploratory) | none (splits 0.0–0.0 %) | none (splits 0.0–0.0 %) | none (splits 0.0–0.0 %) |
| microsoft-decision-1 (exploratory) | 80.7 % (bound 4.6 %, met; splits 80.2–82.7 %) | 72.3 % (bound 4.6 %, met; splits 40.5–73.7 %) | 30.1 % (bound 4.9 %, met; splits 31.5–60.6 %) |
| strands-decider-2b (exploratory) | 3.9 % (bound 14.4 %, not met; splits 0.0–0.0 %) | none (splits 0.0–0.0 %) | none (splits 0.0–0.0 %) |
| unsloth-decision-4b (exploratory) | none (splits 0.0–4.8 %) | none (splits 0.0–0.0 %) | none (splits 0.0–0.0 %) |
Bound: the one-sided 95 % upper bound on the held-out errors of both halves pooled, rounded up; approximate, since each half's threshold depends on the other half; computed after the study from its published counts, not pre-registered (#213). Met: that bound is within 5 %; only a share that meets it is called validated. Each met cell is a separate statement, each at 95 %, approximate; read together they do not all hold at 95 %, so no claim is made about all of them at once, and none should be picked as the best. None: no threshold was validated on either half at seed 2026. Splits: the same share over split seeds 2027–2036; that range need not contain the seed-2026 share, and a share far outside it rests on one split. The shares describe these 2,974 translated public items, with labels carried over by localisation, not your traffic or future traffic; judges marked exploratory are in no pre-registered test. No ranking: rows follow the study's roster. The shares, their ranges, every other figure with its intervals and the cost of each run are in the v0.6 findings in full; the bounds are computed here from their counts.
v0.5 findings
In short. Asking Qwen3-8B and gemini-3.6-flash several times and counting the votes ranked their errors worse than their own verbalized number in 3 of 4 tests under the pre-registered rule (4 of 4 under the re-reading), and one verdict (H2 for gemini-3.6-flash) changes with how an answer that copies an option's description is scored. On BANKING77 at a 5 % error target, validated on held-out texts, Jev native probability can decide 41.0 % (27.0 %–45.1 % over split seeds 2027–2036) of the texts alone, gemini-3.6-flash verbalized 14.3 % (7.6 %–30.0 %), Qwen3-8B token log-probability none (0.0 %–4.4 %). Caveats: public datasets, probably seen in pretraining; one run each, except Jev and Qwen3-8B verbalized on BANKING77 (3 runs); label noise not measured; the held-out slice not run (#106). The number to act on is the same audit on your own decisions.
At a glance: BANKING77, 3,080 human-labelled banking queries: the share each judge decides alone at a 5 % error target, validated on held-out texts.
| Judge and confidence | Validated share at a 5 % error target | 95 % upper bound on the error rate | Range over split seeds | At a 10 % error target |
|---|
| Jev, native probability | 41.0 % | 3.9 % | 27.0–45.1 % | 72.3 % (bound 9.2 %) |
| gemini-3.6-flash, verbalized | 14.3 % | 2.7 % | 7.6–30.0 % | 45.5 % (bound 5.3 %) |
| Qwen3-8B, token log-probability | none | — | 0.0–4.4 % | 12.5 % (bound 7.2 %) |
Validated: the threshold is chosen on one half of the texts and applied to the other, both ways, and the 95 % upper bound on the held-out error rate is within the target. The bound is the one-sided Clopper–Pearson bound on the held-out errors of both halves pooled, rounded up; it treats the two held-out halves as independent draws, an approximation since each half's threshold depends on the other half. A share observed on all rows without a bound is not a basis for deployment.
Teams are shipping judgment models — TypeSafe's Jev, LLM-as-judge, guardrails, routers — that return a confidence with each decision. The literature studies calibration (Guo et al. 2017; Shao 2026; Huang et al. 2026); what a team needs before automating is the same measurement on its own decisions. judge-audit runs any judge in shadow mode against decisions your humans already made and answers the four questions that matter before you automate:
| Question | Metric | Why a buyer cares |
|---|
| When it says 80 % confident, is it right 80 % of the time? | ECE, reliability diagram | A confident-and-wrong judge automates its own mistakes |
| What share of the work can it decide alone at a bounded error? | safe automation rate, accuracy-coverage curve | The ROI number — with its error bound, the threshold to deploy, and the labels behind it |
| What does it really cost, and how bad is the latency tail? | $ per decision, p50 / p99 | The demo is cheap; the tail is what pages you |
| Has it drifted since last week? | judge-audit check CI gate | Vendors update models without telling you |
The study's design, the eight confirmatory tests and every caveat: v0.5 findings in full.
Quickstart
pip install kunko-judge-audit # every release ships Sigstore-signed build provenance
judge-audit run examples/email-routing/labels.jsonl --judge simulated --target 0.10 # no API key needed
judge-audit check examples/email-routing/labels.jsonl --judge simulated --baseline audit-result.json --target 0.10 --min-safe-rate 0.10:0.60
The start of the run line, SIMULATED:
SIMULATED — not a real vendor audit · judge=simulated n=200 accuracy=85.5% [75.1, 93.7] safe_automation@10%=not_met(75.0% bound_95_heldout_approx=16.6%) all_rows=65.5% …
simulated is a seeded simulator so you can see the whole pipeline in ten seconds; every report it touches is stamped SIMULATED, and its figures say nothing about any real judge. safe_automation@10% leads with the held-out check at a 10 % error target: not_met, because the one-sided 95 % upper bound on the held-out error rate (16.6 %, approximate: it pools the two held-out halves) is above the target, so the 75.0 % held out is not validated. A share is shown as validated only when that bound is within the target. all_rows (65.5 %) is the share the threshold to deploy covers on every labelled decision, the share check --min-safe-rate gates. At the default target, 5 %, the simulator reaches no threshold and the line reads safe_automation@5%=none. check --min-safe-rate 0.10:0.60 exits 0 here and exits 1 if that share falls below 60 %. To audit a real vendor, see docs/real-audits.md.
In the GitHub Action the same controls are inputs: target (the error rate the safe automation rate is headlined at, default 0.05), segment-by (where to look for the worst segment: label, meta.FIELD or none) and min-safe-rate (RISK:SHARE minimums that fail the job). It returns safe-automation-rate (at the target) and safe-automation-rates (every target) as outputs.
What you get
One self-contained report per run (Markdown or HTML, prints to A4): the answer first: the share validated on held-out decisions, with its bound and its range over split seeds, then the threshold to deploy; calibration with intervals; accuracy against coverage; cost and latency; the provenance of every number. Shown here on the seeded simulator, stamped SIMULATED. audit-result.json carries the same numbers for your pipeline.
How it works
- Labeled dataset — JSONL rows
{state, questions, labels}. Your humans already decided; the judge must reproduce them. An optional first line declares where the labels come from — their ground-truth tier, from GT-1 constructed to GT-6 production outcome, and what each tier lets you claim.
- Shadow run — the judge scores every row. Nothing is automated; everything is recorded: decision, confidence, latency, cost, raw response.
- Audit report — calibration (ECE with equal-width and with equal-mass bins, Brier score, reliability bins — three numbers, plus log loss next to Brier, never combined), selective prediction (accuracy at every coverage level), cost, latency percentiles, plus provenance: model, backend, timestamp, dataset hash. Accuracy, the three calibration numbers and zero-error coverage carry a 95 % bootstrap interval over the dataset's distinct texts (Jev on the bare-label router:
66.7% [52.5, 80.3]; 2,000 seeded resamples, pure Python, --no-ci to skip), or the exact binomial interval where the bootstrap cannot move so a number on 120 rows reads as the noisy estimate it is. Every report header carries a Ground truth: GT-1 constructed — … line with the tier's meaning and the dataset's caveats, so two accuracies are never read as the same evidence; a dataset that declares nothing is reported as GT-0 unknown, never silently.
- CI gate —
judge-audit check --baseline baseline.json --max-ece-drift 0.02 fails the build when the judge degrades.
Exit codes: 0 ok · 1 drift detected · 2 usage or configuration error, or a judge that could not answer (an HTTP error, an expired key, a server that cannot be reached; one line, judge request failed: …, no traceback). The message says what to fix. An output that cannot be written exits 2 even when drift was also found: fix it before reading the drift.
In CI: the GitHub Action runs the audit on every push or pull request and fails the build on drift:
- uses: kunko-ai-labs/[email protected] # or pin the release's commit SHA
with: { labels: audits/labels.jsonl, judge: jev, baseline: audits/baseline.json,
target: "0.05", segment-by: label, min-safe-rate: "0.05:0.40" }
env: { AI_GATEWAY_API_KEY: ${{ secrets.AI_GATEWAY_API_KEY }} }
One sticky PR comment that opens with the safe automation rate validated on held-out decisions, then the audit table (n, accuracy, ECE, cost, p99, verdict vs baseline) and, labelled as observed on all rows without a bound, the zero-error coverage; the report in the job summary, the report and result JSON as an artifact. The per-decision judgments stay on the runner unless you ask for them (upload-evidence: "true"). min-safe-rate: "0.05:0.40" fails the job unless at least 40 % of the labelled decisions is automated at a 5 % error target (inputs in the Quickstart). Outputs accuracy, ece, zero-error-coverage, safe-automation-rate (at the target), safe-automation-rates (every target), drift for anything downstream. Details in docs/integrations.md.
From inside an agent: pip install "kunko-judge-audit[mcp]" then claude mcp add judge-audit -- judge-audit-mcp (or the equivalent in Cursor). The agent gets run_audit, check_drift and list_judges and can audit the judge it is about to rely on without leaving the session. See docs/integrations.md.
Judge interface
Anything that maps (state, questions) -> (decision, confidence) plugs in:
from judge_audit import Judge, Question, Judgment
class MyJudge(Judge):
name = "my-judge"
def decide(self, state: str, questions: list[Question]) -> list[Judgment]:
... # return Judgment(question=q.name, decision="spam", confidence=0.93)
Ships with fourteen: jev (TypeSafe Jev — and, via JEV_ENDPOINT, any Jev-compatible server such as OpenJev), llm (any chat model with a confidence prompt: Claude through the official SDK, anything OpenAI-compatible — OpenAI, Gemini, Ollama, vLLM — or your own transport), nli (a local zero-shot encoder, the control), finetuned (your own classifier, trained on your labels with scripts/train_classifier.py), laya (Laya, an open-weight judgment model run locally), decision2 (Decision 2.0, an open decision model run locally at a pinned revision), decider (decider, likewise), strands (Strands Decider, likewise), clef (Clef and Clef-flash, likewise), openai-decisions (the OpenAI Decisions API, hosted), clef-hosted (Clef and Clef-flash on Workers AI, hosted), microsoft-decision (Microsoft-Decision-1 through OpenRouter, hosted), logprob (an open model's own probability of each option, run locally with MLX) and simulated. Details in docs/judges.md.
Learn more
Why calibration, not accuracy? Accuracy tells you who wins a benchmark. Calibration tells you what you can automate safely. A judge that is confident on the decisions it gets wrong automates its own mistakes; one that flags its own doubt hands those to people. Audit the honesty, not the score.
FAQ
Is this a benchmark? No. A benchmark ranks judges on a fixed test. An audit checks one judge on your decisions and tells you what you can automate. The datasets here are worked examples, not a leaderboard — yet.
Why was Jev audited first? Its vendor sells calibrated confidence as the headline feature. A claim that specific deserves an independent check.
Can I trust a 100 % result? Only as far as the dataset. The clean-email audit says the judge handles templated business email; it says nothing about your inbox. That is why the adversarial and routing audits exist.
Does it send my data anywhere? Only to the judge endpoint you configure. Reports and checkpoints are local files; commit them or not. In CI the Action uploads the report and the metrics JSON as a run artifact; the per-decision judgments are uploaded only if you set upload-evidence: "true".
License
Apache-2.0 — see LICENSE. Sister project: agent-assurance (deterministic checks on what an agent may do; this repo is the probabilistic half: whether its judgment can be trusted).
