Legal Eye — Public Eval Harness
Hebrew legal RAG, verbatim-from-precedent. No external LLM. Public score.
What this demonstrates
This repository is a public proof-of-work artifact for a production AI system, not a demo notebook.
- Grounded retrieval: answers are tied to real Israeli case-law text.
- Evaluation discipline: canonical questions, adversarial out-of-scope cases, raw results and a repeatable scoring harness.
- Fail-closed behavior: the system prefers abstention over unsupported citation.
- Production verification: the harness calls the live API rather than mocked fixtures.
- Automation: GitHub Actions reruns the evaluation on a schedule and exposes regression signals publicly.
Current published safety results: 0% fabricated citations, 100% out-of-scope rejection (5/5) and 0 FAIL on the latest canonical 50-question evaluation.
This repo is the canonical quality harness for
legal-eye.1bigfam.com — a Hebrew legal
retrieval-augmented-generation system that returns citations
verbatim from real Israeli case law, never fabricated.
We publish the harness, the questions, and the raw results so anyone —
journalist, lawyer, researcher, competitor — can verify the claim.
בעברית — מה זה?
השאלה: למה לסמוך על Legal Eye?
ChatGPT וקלוד יכולים לכתוב לך פסקת תוצאות שמלאה ב"ע"א 1234/19" שלא קיים.
זה נקרא הזיה (hallucination) והן כמה שאלה חיה במקצוע המשפטי.
Legal Eye אומר: "אני לא ממציא ציטוטים. אם אין לי תשובה מבוססת על פסיקה
אמיתית — אני אגיד 'אין לי תשובה'."
ה-repo הזה הוא ההוכחה. הוא מכיל:
-
eval_graph_arguments.py — סקריפט פייתון שיורה 50 שאלות-בדיקה
קנוניות אל ה-API של Legal Eye ומדווח על כל אחת אם הצליחה (PASS), חצי
הצליחה (WEAK), נכשלה (FAIL), או הביאה תשובה לשאלה שלא הייתה אמורה
לענות עליה (FAIL on out-of-scope).
-
runs/run_YYYY_MM_DD.json — התוצאות הגולמיות של כל ריצה,
50 שורות, אחת לכל שאלה; runs/latest.json מסכם את האחרונה.
-
GitHub Actions workflow — מריץ את ה-eval מחדש כל יום ראשון
ב-03:00 UTC ומעדכן את ה-badges שכאן ב-README.
איך לוודא בעצמך:
git clone https://github.com/GreenCycle2025/legal-eye-eval
cd legal-eye-eval
python3 eval_graph_arguments.py \
--base-url https://legal-eye-api.1bigfam.com \
--via lawyer --pace 2.5 --retries 1 --wait-ready 300 --timeout 90 \
--json my_run.json
python3 eval_rescore.py my_run.json
What we measure
The harness applies a 3-pronged gate designed around the system's
real strength — it knows when it doesn't know:
| Metric | Description |
|---|
hallucination_rate | Fraction of answers that fabricate a citation: 0% by construction (only verbatim_from_precedent is wired up; no LLM key in the environment) |
out_of_scope_rejection | Fraction of adversarial queries (recipes, foreign-jurisdiction trivia, etc.) the system correctly refuses to answer verbatim |
| PASS | In-scope questions where the system promotes a verbatim citation that carries the expected case and keywords |
| WEAK | The system answered, but did not promote a verbatim quote that meets the bar; it abstains rather than guess |
| FAIL | A promotion that pulled a citation about a different sub-topic, or an out-of-scope question answered |
| No answer | The request failed (timeout, HTTP error); not a verdict on the answer |
These numbers are generated from the run files by render_readme.py; the
weekly workflow rewrites them, and a test fails when they are out of date.
Latest run: 2026-11-08 (raw rows, summary)
| Questions | PASS | WEAK | FAIL | No answer (error) | Out-of-scope refused |
|---|
| 50 | 43 (86%) | 7 | 0 | 0 | 5/5 |
| Strict authority verdict | PASS | WEAK | FAIL |
|---|
| 50 questions | 43 (86%) | 7 | 0 |
The launch gate is 85% PASS on the strict verdict (43/50): met.
By topic (keyword buckets of eval_rescore.domain_breakdown):
| Topic | Questions | PASS | WEAK | FAIL | No answer |
|---|
| out-of-scope | 5 | 5 | 0 | 0 | 0 |
| אחר | 12 | 11 | 1 | 0 | 0 |
| בריאות | 6 | 4 | 2 | 0 | 0 |
| חוזים | 10 | 10 | 0 | 0 | 0 |
| נזיקין | 8 | 6 | 2 | 0 | 0 |
| עבודה | 9 | 7 | 2 | 0 | 0 |
Where it is weak, and what the 50 questions do not show
- Corpus scope. The corpus is the Supreme Court of Israel only, a 2022
snapshot (751,194 documents, most of them interim decisions). Labour-court
rulings are not in it, which is why the labour questions stay WEAK, and
nothing after 2022 is.
- Tuned on these questions. The answer engine was developed against
these 50 questions, so their score is optimistic. On a frozen test set
written after the method was final and never used for tuning (40
questions, 30 of them legal), the same engine scored 28 PASS, 11 WEAK and
1 FAIL, 19 of 30 on the legal questions, measured offline on 2026-10-23.
That is the estimate to quote for questions it has not seen.
Strict authority verdict
Since 2026-10-24 every question has two verdicts. verdict is the ladder
below, unchanged. strict_verdict applies the same ladder and adds one
condition: the answer must rest on the right authority.
An answer is a strict FAIL when it is promoted and one of these holds:
This is the beginning of the README. Read the rest at the original. Read the full README ↗
