LAIBench is a technical benchmark framework — not a medical device, not regulatory approval, and not clinical validation. It must not be used as the sole basis for clinical deployment decisions. Everything below is within that technical scope.
Why LAIBench
Why LAIBench exists?
Existing LLM benchmarks measure general reasoning. Radiology requires evaluation of clinical safety, report faithfulness, terminology consistency, and critical finding detection. LAIBench was created to measure those properties.
Most "report quality" benchmarks reward fluent prose. A confidently-written report that misses a pneumothorax can score well. LAIBench is built so that cannot happen:
- Hard critical-finding veto. A missed or fabricated critical finding caps the score and forces
FAIL — no matter how polished the rest of the report is. Form never rescues substance.
- No prose/aesthetic axis. There is no standalone style, fluency, or "communication quality" dimension. The only discourse-adjacent signals (hedging, verbosity) are minor, non-gating, and apply only on the no-gold fallback path.
- Conservative combination. The combined dimension score is
MIN(deterministic, judge) — an optional LLM judge can lower a score but never inflate past the deterministic gate.
- Anti-compensation. Strong terminology/coverage cannot lift a clinically weak report into a PASS band.
- Tamper-resistant numbers. Every loaded run is re-scored through the gated combiner; a run whose stored verdict disagrees with the re-derived one is rejected, so a relabeled critical-miss can never inflate the public leaderboard.
- Source-anchored gates without answer keys. On the input-only path the input findings are the evidence: every critical they affirm must survive with the same polarity and side (C09), no critical may be invented (C10), abnormal findings must be reflected (Q16), and an abnormal study cannot be reported as normal (Q17).
- Provenance. A
scoringHash (over the gate-deciding source files) and a cases-only suiteHash pin exactly what produced each number.
Pure TypeScript, zero runtime dependencies, deterministic checks, optional frozen judging, bootstrap CIs, McNemar / Cohen's h.
What it evaluates
LAIBench evaluates reporting behavior from provided text evidence: whether a system can convert an exam descriptor and concise findings into a faithful radiology report under the public contract. It is not primary image interpretation.
It makes failure modes visible: clinically relevant omissions, hallucinated/unsupported findings, factual contradictions, critical-finding preservation, structured-report compliance, terminology and report-language compliance, privacy hygiene, and auditability.
Scoring
| Dimension | Weight | Purpose |
|---|
| CRIT | 30% | Critical-finding preservation and unsafe-negation checks |
| QUAL | 25% | Clinical quality, finding preservation, hallucination resistance, impression synthesis |
| TERM | 20% | Locale, modality, section, terminology, and report-language contract |
| GUIDE | 15% | Guideline and anatomical coverage expectations |
| RAG | 10% | Evidence fidelity, section order, laterality, levels, and measurements |
Critical-finding omissions, unsafe negations, contradictions, unsupported normalcy, wrong report language, and structural errors trigger failure gates — a high average score cannot hide them. See docs/laibench-leaderboard-methods.md.
Quickstart
npm ci
npm run typecheck
npm test
npm run guard:public # public-release safety gate
npm run smoke:mock # run the synthetic en-US suite with a bundled mock agent
npm run smoke:leaderboard # build a leaderboard from the run
npm run smoke:reference # template reference agent on the en-US suite (31 PASS / 18 PARTIAL / 0 FAIL)
npm run ladder # public discrimination ladder → site/ladder.json + docs/public-ladder.md
Does the score fall when the report gets worse?
npm run ladder degrades a template reference report by exactly one error per rung on the 49 public cases — no answer keys needed — and scores every version with the real harness. Under v3.11.0 every critical rung (critical denied, omitted or fabricated, and a "No acute abnormality." null report) is vetoed, and no critical rung passes. Full table: docs/public-ladder.md.
Run the synthetic public en-US suite with a local command adapter:
npm run bench -- suite \
--suite suites/lite-public.en-US.json \
--provider command \
--cmd "node examples/mock-agent.mjs" \
--run-name mock-agent \
--track agent \
--out runs/mock-agent.json
Benchmark your own agent
Any agent that reads a JSON job on stdin and writes the report HTML to stdout works with the command provider — no SDK required:
npm run bench -- suite \
--suite suites/lite-public.en-US.json \
--provider command \
--cmd "node path/to/your-agent.mjs" \
--run-name my-agent --track agent \
--entity-name "My Lab" --entity-type org --system-type product-agent \
--out runs/my-agent.json
examples/mock-agent.mjs is a minimal reference adapter. OpenRouter and any OpenAI-compatible endpoint are also supported (--provider openrouter / --provider openai-compatible); examples/smoke-openai-compatible.mjs shows the shape.
Reliability (pass^k)
npm run bench -- reliability \
--inputs runs/run-1.json runs/run-2.json runs/run-3.json \
--out runs/reliability.json --markdown runs/reliability.md
Frozen predictions
npm run bench -- eval-submission \
--suite suites/lite-public.en-US.json \
--predictions predictions/my-agent.jsonl \
--run-name my-agent --track agent --out runs/my-agent.json
Each JSONL line follows the prediction record schema (instance_id, model_output). See docs/public-submissions.md.
This is the beginning of the README. Read the rest at the original. Read the full README ↗
