llm-rubric-eval

Score LLM answers on verifiable correctness, and keep fluency as a separate signal, so a confident-but-wrong model can't win your evaluation.

Why this exists

When I chose the model behind NYU Stern's V-Lab research assistant, I benchmarked six LLM providers on a 40-point rubric covering accuracy, depth and reasoning. Partway through, I found the rubric couldn't separate a correct answer from a fluent one. Two of the three criteria were judged from how an answer read, so confident phrasing was earning points that correctness should have earned. I had to rebuild the rubric and re-score everything.

This tool makes that failure impossible to miss:

• Correctness comes only from verifiable checks: required facts, numbers within a tolerance, regex patterns, and penalties for known-wrong claims.
• Fluency and confidence are measured as well, but they never score points. They are reported on a separate axis.
• Every answer is plotted on style vs substance. Answers that sound sure but score low are flagged as confidently wrong.
• Each model is ranked twice, once by correctness and once by a "naive" style-weighted rubric. If the rankings disagree, the report says so.

In the bundled sample run, the confident model is last on correctness (0.14) but would be #2 under a style-weighted rubric.

Quick start

git clone https://github.com/mgupta646/llm-rubric-eval && cd llm-rubric-eval
pip install -e ".[dev]"

# score the bundled sample answers and write an HTML report
rubriceval score --questions examples/questions.json --answers examples/answers --out report.html

| Rank | Model                | Correctness | Fluency | Naive score | Confidently wrong |
|---:|---|---:|---:|---:|---:|
| 1 | sample-a (accurate)    | 1.00 | 0.71 | 0.80 (#1) | 0  |
| 2 | sample-c (hedged)      | 0.95 | 0.34 | 0.55 (#3) | 0  |
| 3 | sample-d (mixed)       | 0.60 | 0.50 | 0.53 (#4) | 0  |
| 4 | sample-b (confident)   | 0.14 | 0.96 | 0.69 (#2) | 11 |

! A style-weighted rubric would rank these models differently.

The four sample "models" are hand-written synthetic answers designed to show distinct behaviours (accurate, confident-but-wrong, hedged, mixed). They are not outputs of real models; see examples/make_sample_answers.py.

Score real models

export GROQ_API_KEY=...            # or OPENAI_API_KEY / ANTHROPIC_API_KEY
rubriceval collect --questions examples/questions.json --provider groq --out answers/groq.jsonl
rubriceval collect --questions examples/questions.json --provider anthropic --out answers/claude.jsonl
rubriceval score --questions examples/questions.json --answers answers --out report.html

Keys are read from environment variables only. They are never written to disk or passed as arguments.

Writing a rubric

Each question carries its own criteria. Points come only from checks you could verify by hand:

{
  "id": "q02_var",
  "question": "A portfolio's 1-day 99% Value at Risk is $1 million. What does that mean?",
  "criteria": [
    {"id": "tail_probability", "type": "any_of",   "points": 1, "any_of": ["1%", "99%"]},
    {"id": "exceedance",       "type": "any_of",   "points": 1, "any_of": ["exceed", "more than", "greater than"]},
    {"id": "about_losses",     "type": "regex",    "points": 1, "pattern": "\\bloss"},
    {"id": "not_a_max",        "type": "forbidden","penalty": 1.5, "phrases": ["maximum loss", "worst case"]}
  ]
}

Type	Passes when	Notes
any_of	any listed phrase appears	case, accent and dash-insensitive
all_of	every listed phrase appears
number	a number in the answer is within abs_tol (or rel_tol) of value	parses $1,157.63, 12%, -3.5
regex	the pattern matches the normalized answer
forbidden	none of the known-wrong claims appear	a hit subtracts penalty

forbidden matters more than it looks. In the sample, one answer swaps precision and recall but still contains both correct formulas. Only the forbidden-claim check catches it.

How it works

questions.json ─┐
                ├─► checks.py   verifiable criteria → correctness (0–1)
answers/*.jsonl ┤
                ├─► fluency.py  length, sentence shape, assertive vs hedging words → fluency, confidence
                │
                └─► scorer.py   flags: confidently_wrong · hedged_but_right · known_wrong_claim
                                leaderboard: rank by correctness vs style-weighted "naive" rank
                                      └─► report.py  self-contained HTML (inline SVG, no JS)

Limitations

• Phrase and number checks are only as good as the rubric. Write forbidden entries for the mistakes you expect.
• Fluency and confidence are transparent heuristics, not a language model. They exist to expose style, not to grade it.
• Open-ended questions with many valid phrasings need broader any_of lists, or a human pass.

Development

pytest -q        # 20 tests

---

Built by Mehul Gupta · Portfolio · LinkedIn