contract-review-eval-harness
Find out whether an AI contract review is actually good: scores clause coverage, risk severity and citation grounding against a hand-authored gold set, and rejects fabricated citations.
Offline, deterministic, no API key. Synthetic contracts only. Not legal advice.
Actual output of make demo, rendered as an image.
Quick start
Needs uv, which fetches a supported Python if none is installed.
git clone https://github.com/sebastianfoerste/contract-review-eval-harness
cd contract-review-eval-harness
uv sync
uv run python -m contract_eval evaluate --case nda # writes scorecard.md
Score your own tool's review JSON (schema) against a bundled contract:
uv run contract-eval score my-review.json --case nda # exit 1 on REJECT
Example output
Selected rows from the NDA scorecard of the offline stub, which is deliberately imperfect:
| Dimension | Score | Notes |
|---|
| Clause F1 | 0.96 | harmonic mean of precision and recall |
| Risk precision | 0.83 | flagged clauses that the gold set treats as risks |
| Severity accuracy | 0.60 | correct severity among jointly identified risks |
| Citation grounding | 0.89 | 8/9 quotes matched as an exact span of the normalised source |
| Hallucination count | 1 | cited quotes not grounded in the source |
The one fabricated citation is a hard REJECT.
Links
- Overview: why this exists, what it checks, more commands, how it compares, limitations
- Metric reference
- Legal engineering explainer: the contracts, gold sets, metrics, design principles and likely interview questions
- Score your own tool and gate CI on it
- Release certificate and robustness
- Live runs, capture and replay and the leaderboard
- Results site and playground
- Case study, changelog, contributing
Limitations: synthetic contracts, one annotator's gold sets and three contract types; every flagged risk and citation needs a qualified lawyer before reliance (details).
