Bellwether

A model that notices it is getting worse, and replaces itself.

Bellwether scores English Wikipedia edits for the probability they will be reverted, publishes each score before the outcome exists, grades itself when the outcome arrives, and — when its own accuracy decays — retrains, shadow-tests and replaces its own model without human instruction, recording the evidence for every decision in a public, append-only log.

Live status · Performance · Model timeline · About

---

What it found

The full answer is in the decision memo — two pages, no technical background needed. The short version:

• The system works; the model is ordinary. 65,000+ edits collected, 48,000+ forecasts committed before their outcomes existed, running unattended since 10 August.
• Most of the model's advantage comes from one input. Remove the feature describing how new an account is and the margin falls from 0.107 to 0.039 — below the threshold it must clear. Measured, published, and deliberately not acted on.
• The live accuracy figure does not exist yet. An outcome is only settled after seven days. Publishing a provisional one would be the exact failure the design refuses.
• Its own guarantees caught its own mistakes, repeatedly — a revert rate wrong by 16 points, an accuracy sample that was 100% positive, three deployments that failed while the health check said everything was fine.

This project does not claim its model is good. The claim is that the system can be trusted about the model.

---

Why it exists

A ranking model deployed against a live stream decays from the moment it ships, because the behaviour it ranks is generated by people whose behaviour changes. Almost every deployed model is evaluated once, before launch, against a held-out sample of the past — and thereafter trusted. Nobody measures whether it is still right, nobody defines in advance what "no longer right" means, and the decision to replace it is made by someone who has already seen which replacement won.

Can a system detect its own decay, replace its own model, and be trusted to have done so honestly?

---

How it stays honest

Commits before it knows	Every score is written to a register the writer role cannot UPDATE or DELETE — enforced by database grant, not by careful code
Waits for the answer	A prediction enters a published figure only once its outcome has actually settled, by a rule requiring both elapsed time and an observation
Decided in advance	PREREGISTRATION.md fixed the promotion rule, thresholds and statistical test before the first model existed. The constants live in code and are asserted against that document by the test suite
Publishes refusals	Every rejected challenger appears with the same evidence a promotion would carry. A log holding only changes cannot answer what was refused
States the bad news	The ablation finding, the twelve inert features, and the expectation that Wikimedia's own model wins — all published, all recorded before the results were seen

---

Architecture

MediaWiki API ─▶ ingest ─▶ score ─▶ append-only register
                    │                      │
                    ▼                      ▼
                 labels ──────────────▶ metrics ─▶ triggers ─▶ retrain
                                                        │         │
                                                        ▼         ▼
                                                   decisions ◀─ shadow

Layer	Choice	Why
Pipeline	Python 3.13, scikit-learn	—
Database	Neon PostgreSQL	Append-only enforced by REVOKE; row-level security for the app
Scheduler	GitHub Actions	The only option unlimited and free on public repositories
API	FastAPI on Render (free)	Holds a read-only role; a separate role for the app that cannot touch evidence
Frontend	Next.js on Vercel	—

Total infrastructure cost: zero. Not an incidental property — a constraint that shaped the design, including the parts that are worse for it.

Running unattended

Job	Cadence
Ingest + score	every 10 minutes
Label harvest	every 30 minutes
Metrics	every 6 hours
Reconcile · Reproduce · Lift Wing · Triggers · Decide · Agreement	daily
Watchdog	hourly
Seal + retention	monthly

---

Documents

DECISION_MEMO.md	What the system found. Two pages, non-technical
METHODS.md	Every statistical procedure, and every exclusion with the direction of its bias
PREREGISTRATION.md	The rules, fixed before any model existed
Bellwether_SRS_v1.0.md	The full specification, with four dated amendments
Bellwether_M*_Spec.md / _Summary.md	Per-milestone specifications and findings

---

Public API

No authentication required.

/health       liveness, and which migrations the database has
/stats        ingestion, coverage gaps, matured revert rates
/register     predictions, scoring lag, reproducibility
/metrics      live figures per population, every one with its n
/kc2          the offline backtest, its margin, and the ablation
/calibration  reliability bins, raw and population-weighted
/decisions    every promotion, rejection and rollback with its evidence
/agreement    how good a proxy "was reverted" is for "was a bad edit"

---

Limitations

Stated here rather than buried.

• The live accuracy figure does not exist yet. Seven-day maturity; the register is five days old.
• The model's margin rests substantially on one feature. Published above.
• Twelve of twenty-eight features measure zero importance. Recorded, not removed — removing them after seeing which scored badly is selection.
• Monitoring is a failing scheduled job. That notifies one person by email from GitHub. It is not an on-call system.
• One reviewer exists. Human-judgement figures describe one person's opinion until there are two.
• The free tier sleeps. First page load can take up to a minute.
• Reproducibility is claimed for the serving model only. A correctness fix changed how state is derived, so earlier predictions genuinely cannot be re-derived — reported in its own column rather than counted as success or failure.
• It reads a sample, not the whole feed — 50% of logged-out edits, 3% of registered — corrected by weighting everywhere.
• It never writes to Wikipedia.

---

Development

python -m venv .venv && .venv/Scripts/pip install -r requirements-dev.txt
python scripts/bootstrap_database.py "<owner-connection-string>"
pytest -q && mypy bellwether && ruff check . && ruff format --check .

449 tests. The suite includes the append-only guarantees, which are checked by connecting as the restricted role and attempting the forbidden action — a grant cannot be tested against a mock.

---

Author: Muhammad Haris Khokhar · Source