Microcosm calibration diagnostics

Interactive dashboard for PolicyEngine's calibrated Microcosm synthetic populations for the US, UK, and Belgium. Each country is published as its own Hugging Face dataset; the US repository is public, while the UK and Belgium repositories require a server-side Hugging Face token.

Deprecated upstream identifiers: Microcosm's current Hugging Face slugs, release/file names, deployment variables, and Chronicle wire contracts still use populace/ledger; see upstream identifier compatibility.

Release metadata and diagnostics are read from exact Hugging Face commits. The calibration explorer instead downloads a precomputed tree bundle from private Vercel Blob storage through a public, same-origin Next.js API route. It renders the roots first and then loads deeper hierarchy levels in increasing depth order. A dashboard-owned manifest maps latest to an immutable release and commit. There is no committed data snapshot; the Next.js route handlers are the dashboard API layer.

What it shows

• Release summary (/microcosm) — calibration fit KPIs, target coverage, per-family fit, and worst-fit / biggest-improvement targets, for the current release (or any published release via ?release=). Staging candidates appear only on the Staging candidates page. An identifier that publishers label differently (HMRC "Hartlepool UA", ONS "Hartlepool") is reported, not refused: the first spelling is shown and the count surfaces on the page.
• Target diagnostics (/microcosm/targets) — browse the calibration target surface by the quantity each constraint measures (e.g. adjusted gross income), then drill its breakdown dimensions (income band x return type x filing status, geography, ...). Every axis a variable varies on becomes a filterable, sortable facet, and any target opens a canonical detail card (structured registry fields, source citation, initial -> final -> target).
• Compare versions (/microcosm/compare) — diff two releases: targets matched by name, common targets get a fit change, and added/removed targets are surfaced.
• Staging runs (/microcosm/staging) — monitor pre-release Microcosm build runs from a country's staging Hub repo: current stage, calibration loss progress, final candidate diagnostics once they exist (uploaded with the telemetry, or read from the dataset bundle the run staged under staged/<run_id>/ in the release repository, flagged as an unreleased staged dataset on this page), candidate-vs-current-release fit, and a hierarchical map of weighted target-error increases and reductions. Countries without a staging repository show an explicit unavailable state. A run's status, stage and times come from the hosted telemetry collector when it has a record of the run, because a build's Hugging Face copy can stop updating partway through; the page then names the source and says what the Hugging Face copy last recorded. On both tabs, a collector record that stopped before the run ended gives way to a Hugging Face copy that recorded the end later. Runs only the collector knows have no candidate files and appear on Build progress only.
• Build progress (the second tab of Staging candidates, /microcosm/staging?view=progress, with &run=<run id> for one run) — follow a build run while it runs, from the staging repository or from the runs/<run_id>/ telemetry folders a Microcosm build writes on the machine it runs on: a stage timeline and an expected finish time with a 90% range. See the build progress notes.
• Calibration target investigations (docs/ai/) — tool-independent procedures, specialist review responsibilities, and a reusable checklist for identifying the cause of a discrepant target from release artifacts and relevant source repositories.

API

The Next.js route handlers are the API layer. Mutable staging data reads from Hugging Face; immutable calibration trees and comparisons read through private Vercel Blob storage.

Published-release endpoints accept country=us|uk|be (default us when the parameter is absent). Any other value, including another spelling such as US, returns HTTP 404 with {"error": "country_not_configured", "requested_country", "reason", "detail"}; it is never answered with another country's release. See the request contract.

This is the beginning of the README. Read the rest at the original. Read the full README ↗