LAIBench

---

LAIBench is a technical benchmark framework — not a medical device, not regulatory approval, and not clinical validation. It must not be used as the sole basis for clinical deployment decisions. Everything below is within that technical scope.

Why LAIBench

Why LAIBench exists? Existing LLM benchmarks measure general reasoning. Radiology requires evaluation of clinical safety, report faithfulness, terminology consistency, and critical finding detection. LAIBench was created to measure those properties.

Most "report quality" benchmarks reward fluent prose. A confidently-written report that misses a pneumothorax can score well. LAIBench is built so that cannot happen:

• Hard critical-finding veto. A missed or fabricated critical finding caps the score and forces FAIL — no matter how polished the rest of the report is. Form never rescues substance.
• No prose/aesthetic axis. There is no standalone style, fluency, or "communication quality" dimension. The only discourse-adjacent signals (hedging, verbosity) are minor, non-gating, and apply only on the no-gold fallback path.
• Conservative combination. The combined dimension score is MIN(deterministic, judge) — an optional LLM judge can lower a score but never inflate past the deterministic gate.
• Anti-compensation. Strong terminology/coverage cannot lift a clinically weak report into a PASS band.
• Tamper-resistant numbers. Every loaded run is re-scored through the gated combiner; a run whose stored verdict disagrees with the re-derived one is rejected, so a relabeled critical-miss can never inflate the public leaderboard.
• Source-anchored gates without answer keys. On the input-only path the input findings are the evidence: every critical they affirm must survive with the same polarity and side (C09), no critical may be invented (C10), abnormal findings must be reflected (Q16), and an abnormal study cannot be reported as normal (Q17).
• Provenance. A scoringHash (over the gate-deciding source files) and a cases-only suiteHash pin exactly what produced each number.

Pure TypeScript, zero runtime dependencies, deterministic checks, optional frozen judging, bootstrap CIs, McNemar / Cohen's h.

What it evaluates

LAIBench evaluates reporting behavior from provided text evidence: whether a system can convert an exam descriptor and concise findings into a faithful radiology report under the public contract. It is not primary image interpretation.

It makes failure modes visible: clinically relevant omissions, hallucinated/unsupported findings, factual contradictions, critical-finding preservation, structured-report compliance, terminology and report-language compliance, privacy hygiene, and auditability.

Scoring

Dimension	Weight	Purpose
CRIT	30%	Critical-finding preservation and unsafe-negation checks
QUAL	25%	Clinical quality, finding preservation, hallucination resistance, impression synthesis
TERM	20%	Locale, modality, section, terminology, and report-language contract
GUIDE	15%	Guideline and anatomical coverage expectations
RAG	10%	Evidence fidelity, section order, laterality, levels, and measurements

Critical-finding omissions, unsafe negations, contradictions, unsupported normalcy, wrong report language, and structural errors trigger failure gates — a high average score cannot hide them. See docs/laibench-leaderboard-methods.md.

Quickstart

npm ci
npm run typecheck
npm test
npm run guard:public        # public-release safety gate
npm run smoke:mock          # run the synthetic en-US suite with a bundled mock agent
npm run smoke:leaderboard   # build a leaderboard from the run
npm run smoke:reference     # template reference agent on the en-US suite (31 PASS / 18 PARTIAL / 0 FAIL)
npm run ladder              # public discrimination ladder → site/ladder.json + docs/public-ladder.md

Does the score fall when the report gets worse?

npm run ladder degrades a template reference report by exactly one error per rung on the 49 public cases — no answer keys needed — and scores every version with the real harness. Under v3.11.0 every critical rung (critical denied, omitted or fabricated, and a "No acute abnormality." null report) is vetoed, and no critical rung passes. Full table: docs/public-ladder.md.

Run the synthetic public en-US suite with a local command adapter:

npm run bench -- suite \
  --suite suites/lite-public.en-US.json \
  --provider command \
  --cmd "node examples/mock-agent.mjs" \
  --run-name mock-agent \
  --track agent \
  --out runs/mock-agent.json

Benchmark your own agent

Any agent that reads a JSON job on stdin and writes the report HTML to stdout works with the command provider — no SDK required:

npm run bench -- suite \
  --suite suites/lite-public.en-US.json \
  --provider command \
  --cmd "node path/to/your-agent.mjs" \
  --run-name my-agent --track agent \
  --entity-name "My Lab" --entity-type org --system-type product-agent \
  --out runs/my-agent.json

examples/mock-agent.mjs is a minimal reference adapter. OpenRouter and any OpenAI-compatible endpoint are also supported (--provider openrouter / --provider openai-compatible); examples/smoke-openai-compatible.mjs shows the shape.

Reliability (pass^k)

npm run bench -- reliability \
  --inputs runs/run-1.json runs/run-2.json runs/run-3.json \
  --out runs/reliability.json --markdown runs/reliability.md

Frozen predictions

npm run bench -- eval-submission \
  --suite suites/lite-public.en-US.json \
  --predictions predictions/my-agent.jsonl \
  --run-name my-agent --track agent --out runs/my-agent.json

Each JSONL line follows the prediction record schema (instance_id, model_output). See docs/public-submissions.md.

Website

laibench.laudos.ai is a static site in site/ that runs the real scorer in the browser: scripts/build-site-scorer.mjs bundles src/ with esbuild (Node built-ins shimmed in site-src/shims/) and a test requires the bundle to return the same verdict, score, gate reasons and per-check outcomes as the CLI. The site has the pneumothorax test, a playground that marks up any report with the scorer's evidence, the discrimination ladder, the leaderboard (each row states the scoring version that produced it), the 49 public cases, and in-browser recomputation of the suiteHash and scoringHash.

npm run site:build   # bundle the scorer, copy the public cases, write site/provenance.json
npm run ladder       # regenerate site/ladder.json
npx serve site       # or any static server

npm test fails if site/assets/laibench-scorer.js or site/provenance.json is stale relative to the scoring source.

Calibration controls

The public stratified page is a reference-vs-null sanity check: it verifies the harness is not inverted by comparing public reference reports against a fixed unsafe null baseline. That is a sanity check, not a claim that the null baseline measures realistic model degradation. Dose-response controls require answer-key material and are generated only inside the controlled environment; public releases publish aggregate calibration summaries only — never case-level predictions or gold labels.

Open vs controlled (read this before citing)

Two distinct artifacts exist, and only one is open:

• LAIBench (public). The open, downloadable, openly-reproducible set.
• Controlled pt-BR suite. A gated, aggregate-only suite. Case JSON, answer keys, frozen predictions, and provenance are not distributed here and cannot be reconstructed from this repository. Cite its numbers only as controlled, aggregate-only results locked to a specific suite hash and case count.

Do not attach "open benchmark" language to the controlled suite.

Case provenance

The public demonstration cases under cases/public/ are synthetic and input-only (every case is flagged synthetic: true). The controlled pt-BR cases are synthetic and were authored and reviewed by senior radiologists in São Paulo, SP, Brazil. Synthetic authorship and internal senior-radiologist review are a data-quality process, not an independent third-party validation; external inter-rater adjudication is tracked as future work and is not claimed here. A radiologist-review claim on any public score is gated by npm run guard:public and npm run laibench:validate-adjudication.

Data boundary

This repository ships synthetic, public-safe material only — code, schemas, docs, the site, and a tiny synthetic demo suite. It contains no raw clinical reports, private corpus, hidden test sets, answer keys, private scoring criteria, credentials, or internal tooling. The boundary is enforced by npm run guard:public (and guard:private) in CI. See DATA_ACCESS_POLICY.md.

Governance & contributing

• GOVERNANCE.md — suite tiers (public-smoke / controlled-eval / hidden-holdout / calibration), claim discipline, anti-gaming posture, and the honest roadmap to external-validation grade.
• docs/benchmark-cards.md — per-suite cards: visibility, case source, leakage risk, allowed use, disallowed claims, suite/scoring hashes, adjudication status.
• CONTRIBUTING.md — how to propose scoring/methodology changes.
• SECURITY.md — report vulnerabilities privately.
• CODE_OF_CONDUCT.md.

Citation

@software{laibench,
  title  = {LAIBench: a governance-oriented benchmark for AI-assisted radiology reporting},
  author = {{Laudos.AI Research}},
  year   = {2026},
  url    = {https://laibench.laudos.ai}
}

Companion (conceptual): Beyond Templates: A Compositional Model and Lower Bound for Radiology Report Variability.

License

Released under the MIT License. It applies to the public code, schemas, documentation, examples, synthetic demo cases, and tooling in this repository. It does not apply to the private clinical corpus, gated datasets, hidden test sets, answer keys, private scoring criteria, or protected evaluation artifacts — none of which are included here.