signals-benchmark

An open leaderboard for the task that matters to a foresight product: generating good "signals of change."

Plug in one OpenRouter key. Get a ranked, four-axis leaderboard across 25+ frontier models, scored over a frozen suite of multi-sector briefs. Re-runnable when new models ship — add the slug, run with --into, publish.

Status: alpha. The methodology is opinionated and intentionally open about its limitations — see Known limitations. Feedback, model additions, and methodology critiques welcome via issues and PRs.

Public leaderboard: signals-strict consumes this data at /benchmark.

---

What it measures

Each model is run against a frozen suite of 12 briefs spanning healthcare, fintech, defense, climate, retail, biotech, energy, education, geopolitics, AI infra, mobility, food. Every signal it produces is scored on four orthogonal axes:

Axis	Weight	What it checks	Method
Verifiability	0.40	Is the underlying evidence claim real?	Web-grounded judge classifies into one of 6 buckets: grounded / speculative / future / indicative / dubious / fabricated. The future bucket protects forward-looking foresight from being marked as hallucination.
Specificity	0.30	Named actors? Concrete events? No hype adjectives?	Separate non-grounded judge against an explicit writing rubric — different vendor from the verifier to reduce judge-family bias.
Currency	0.15	How recent is the supporting evidence?	Newest source date from the verifier's annotations, passed through a decay curve.
Coverage	0.15	Breadth across the brief's categories + uniqueness vs. the cohort	Mean of category-balance and unique-share (token-Jaccard < 0.3 vs. every other model's signals on the same brief).

The composite is a weighted mean. Per-axis sub-scores are emitted alongside — if you weight things differently, re-compute from the raw JSON without re-running anything.

Alongside the four axes, a separate probe estimates each model's knowledge cutoff — deliberately not folded into the composite, because it's a property of the model, not a quality judgement.

---

Quickstart

Requires Node 22.6+ (uses --experimental-strip-types, no build step).

pnpm install
cp .env.example .env       # paste your OpenRouter key

Get a key at https://openrouter.ai/keys. ~$5 of credit is plenty to try things out.

# Tiny smoke run — 2 cheap models × 1 brief, ~$0.20, ~1 min
pnpm bench --preset smoke --briefs healthcare-regulated-ai

# Frontier-only on one brief (~$5, ~5 min)
pnpm bench --tiers frontier --briefs healthcare-regulated-ai

# Full top-25 cohort × all 12 briefs (~$15–25, ~60 min)
pnpm bench --preset top25

Output lands in results/<run-id>/:

• meta.json — run config + cost
• runs/<brief>__<model>.json — raw generations
• evals/<brief>__<model>.json — per-signal verdicts + citations
• leaderboard.md / leaderboard.json — final ranking

---

Common workflows

Add a new model to an existing run (cheap!)

When a new model ships:

pnpm bench --models newprovider/new-model --into <existing-run-id>

Reuses every cached run/eval from the existing leaderboard, only generates + judges the new model. Typical cost: $3–5 (vs. $15–25 for a fresh full run).

Resume an interrupted run

Killed phase 2? Switch judges mid-run? Use bench:resume:

pnpm bench:resume --run <id> \
  --judge google/gemini-2.5-flash:online \
  --spec-judge google/gemini-2.5-flash

Skips already-completed (model × brief) pairs. Picks up where it left off.

Re-render after tuning the composite

If you change the weights in src/score.ts, re-score from existing evals — no API calls:

pnpm bench:report --run <id>

Inspect the catalog

pnpm bench:list

Shows all 12 briefs + every model the runner knows about.

---

Web viewer (optional, local)

A small Next.js app under web/ renders an interactive leaderboard locally — sortable axes, per-brief drill-down, per-signal verdicts + citations. Useful while iterating.

cd web && npm install && npm run dev      # http://localhost:3030

Reads from ../results/ directly. Zero API calls of its own. Public consumers (e.g. signals-strict) generally consume the published JSON instead.

---

Knowledge cutoff

Currency measures how recent a model's cited evidence is. Cutoff measures how recent its world model is. A model whose knowledge stops fourteen months ago can't produce a current signal of change no matter how well it writes — so the benchmark measures it, and reports it next to the leaderboard rather than inside it.

The method is adapted from Shrivu Shankar's Exploring Claude/GPT knowledge cutoffs. Three probes per model:

Probe	What it asks	What it gives you
Quiz	8-way multiple choice: which of these real events happened in this month?	Accuracy drops from a plateau to chance past the cutoff. The month of the drop is the estimate.
Self-report	"What month is it?" / "When does your training data end?" × 5 phrasings	What the model believes, and how much that belief moves with the phrasing.
Identity	"What model are you?" × 5 phrasings	Doesn't date anything alone, but a model that thinks it's its own predecessor tells you what it was trained on.

Every wrong answer is a real event

This is the part that took three attempts to get right, and it's the part to preserve if you change anything else.

A quiz item leaks whenever an option can be ruled out without knowing the month. Fabricated wrong answers always leak, however carefully they're written:

This is the beginning of the README. Read the rest at the original. Read the full README ↗