contract-review-eval-harness

Find out whether an AI contract review is actually good: scores clause coverage, risk severity and citation grounding against a hand-authored gold set, and rejects fabricated citations.

Offline, deterministic, no API key. Synthetic contracts only. Not legal advice.

Actual output of make demo, rendered as an image.

Quick start

Needs uv, which fetches a supported Python if none is installed.

git clone https://github.com/sebastianfoerste/contract-review-eval-harness
cd contract-review-eval-harness
uv sync
uv run python -m contract_eval evaluate --case nda   # writes scorecard.md

Score your own tool's review JSON (schema) against a bundled contract:

uv run contract-eval score my-review.json --case nda   # exit 1 on REJECT

Example output

Selected rows from the NDA scorecard of the offline stub, which is deliberately imperfect:

Dimension	Score	Notes
Clause F1	0.96	harmonic mean of precision and recall
Risk precision	0.83	flagged clauses that the gold set treats as risks
Severity accuracy	0.60	correct severity among jointly identified risks
Citation grounding	0.89	8/9 quotes matched as an exact span of the normalised source
Hallucination count	1	cited quotes not grounded in the source

The one fabricated citation is a hard REJECT.

Links

• Overview: why this exists, what it checks, more commands, how it compares, limitations
• Metric reference
• Legal engineering explainer: the contracts, gold sets, metrics, design principles and likely interview questions
• Score your own tool and gate CI on it
• Release certificate and robustness
• Live runs, capture and replay and the leaderboard
• Results site and playground
• Case study, changelog, contributing

Limitations: synthetic contracts, one annotator's gold sets and three contract types; every flagged risk and citation needs a qualified lawyer before reliance (details).