Microcosm calibration diagnostics

Interactive dashboard for PolicyEngine's calibrated Microcosm synthetic populations for the US, UK, and Belgium. Each country is published as its own Hugging Face dataset; the US repository is public, while the UK and Belgium repositories require a server-side Hugging Face token.

Deprecated upstream identifiers: Microcosm's current Hugging Face slugs, release/file names, deployment variables, and Chronicle wire contracts still use populace/ledger; see upstream identifier compatibility.

Release metadata and diagnostics are read from exact Hugging Face commits. The calibration explorer instead downloads a precomputed tree bundle from private Vercel Blob storage through a public, same-origin Next.js API route. It renders the roots first and then loads deeper hierarchy levels in increasing depth order. A dashboard-owned manifest maps latest to an immutable release and commit. There is no committed data snapshot; the Next.js route handlers are the dashboard API layer.

What it shows

• Release summary (/microcosm) — calibration fit KPIs, target coverage, per-family fit, and worst-fit / biggest-improvement targets, for the current release (or any published release via ?release=). Staging candidates appear only on the Staging candidates page. An identifier that publishers label differently (HMRC "Hartlepool UA", ONS "Hartlepool") is reported, not refused: the first spelling is shown and the count surfaces on the page.
• Target diagnostics (/microcosm/targets) — browse the calibration target surface by the quantity each constraint measures (e.g. adjusted gross income), then drill its breakdown dimensions (income band x return type x filing status, geography, ...). Every axis a variable varies on becomes a filterable, sortable facet, and any target opens a canonical detail card (structured registry fields, source citation, initial -> final -> target).
• Compare versions (/microcosm/compare) — diff two releases: targets matched by name, common targets get a fit change, and added/removed targets are surfaced.
• Staging runs (/microcosm/staging) — monitor pre-release Microcosm build runs from a country's staging Hub repo: current stage, calibration loss progress, final candidate diagnostics once they exist (uploaded with the telemetry, or read from the dataset bundle the run staged under staged/<run_id>/ in the release repository, flagged as an unreleased staged dataset on this page), candidate-vs-current-release fit, and a hierarchical map of weighted target-error increases and reductions. Countries without a staging repository show an explicit unavailable state. A run's status, stage and times come from the hosted telemetry collector when it has a record of the run, because a build's Hugging Face copy can stop updating partway through; the page then names the source and says what the Hugging Face copy last recorded. On both tabs, a collector record that stopped before the run ended gives way to a Hugging Face copy that recorded the end later. Runs only the collector knows have no candidate files and appear on Build progress only.
• Build progress (the second tab of Staging candidates, /microcosm/staging?view=progress) — follow a build run while it runs, from the staging repository or from the runs/<run_id>/ telemetry folders a Microcosm build writes on the machine it runs on: a stage timeline, an expected finish time with a 90% range, calibration speed, where build time goes across past runs of the same pipeline, and which checks fail and how much compute had run when they did. See the build progress notes.
• Calibration target investigations (docs/ai/) — tool-independent procedures, specialist review responsibilities, and a reusable checklist for identifying the cause of a discrepant target from release artifacts and relevant source repositories.

API

The Next.js route handlers are the API layer. Mutable staging data reads from Hugging Face; immutable calibration trees and comparisons read through private Vercel Blob storage.

Published-release endpoints accept country=us|uk|be (default us when the parameter is absent). Any other value, including another spelling such as US, returns HTTP 404 with {"error": "country_not_configured", "requested_country", "reason", "detail"}; it is never answered with another country's release. See the request contract.

Endpoint	Purpose
GET /api/microcosm/releases	List published releases (newest first)
GET /api/microcosm?release=<id>	Published-release summary (default: latest)
GET /api/microcosm/tree?country=<code>&release=<id>&part=<part>	Redirect to an exact build artifact ID and stream one immutable tree-bundle part
GET /api/microcosm/tree-manifest?country=<code>	List selectable release and finalized-staging builds and the current release build
GET /api/microcosm/comparison-tree?country=<code>&a=<build>&b=<build>&mode=reported|shared&part=<part>	Build an ordered pair once when absent, then stream its immutable comparison-tree parts
GET /api/microcosm/target-diagnostics?release=<id>&...	Faceted per-target diagnostics for a published release
GET /api/microcosm/target-investigation?target=<id>&release=<id>	Copyable investigation packet for one published target: fit evidence, chronicle metadata, artifact paths, repo searches, and next checks
GET /api/microcosm/compare?a=<id>&b=<id>	Version-over-version diff
GET /api/microcosm/builds?country=<code>&source=local|staging	Every build run from one source as a stage timeline
GET /api/microcosm/builds/run?id=<run_id>&country=<code>&source=local|staging	One run's timeline and completion forecast, with stage-time and gate-failure statistics for its pipeline
GET /api/microcosm/staging/runs	List staging build runs, each with the collector's state when it has a record of the run (state_source, Hugging Face values in hugging_face_*, and collector_note when the collector is unavailable)
GET /api/microcosm/staging/run?id=<run_id>	One staging run's progress, its state as in the run list (state), and its candidate diagnostics: an uploaded calibration_diagnostics telemetry artifact, or the calibration_diagnostics.json of the dataset bundle the run staged under staged/<run_id>/ in the country's release repository (read at the commit the run recorded, digest-verified)
GET /api/microcosm/staging/target-diagnostics?id=<run_id>&...

/api/microcosm/staging/compare resolves release=latest, its default, for each candidate:

• Local-area candidates. A run is a local-area candidate when the pipeline.id its run_manifest.json records is one of the country's staging.local_area_pipelines in the country registry. The UK lists uk-local-candidate; no other country lists any, so their run manifests are not read for this. A local-area candidate is compared with the most recently published local-area build: among release builds in the tree manifest whose release manifest's dataset_role names a local area (such as non_default_local_area), the one with the latest recorded createdAt, else updatedAt. The local-area release has no pointer file and is never promoted, so it has no current build.
• Every other candidate, and a local-area candidate when the tree manifest lists no local-area release build, is compared with the country's current release. That includes a local-area release published on Hugging Face whose calibration tree the dashboard has not built yet.

Only a local-area candidate's response adds comparison_release, which names the release it was compared with:

Field	Meaning
release_id	The release compared with, the same as a.release_id
basis	newest_local_area_release, or current_release when the tree manifest lists no local-area release build
local_area_candidate	Always true when the field is present

An explicit release=<id> is compared as requested and adds no comparison_release. Responses for US, Belgian and UK national candidates are unchanged.

The staging target-change route supports two comparison modes. reported uses each release's actual target weights and complete target surface, including added and removed targets. shared restricts the calculation to shared targets, normalizes each release's weights over that shared set, averages the two shares target by target, and applies the resulting pooled weights to both releases.

Version and staging comparisons use the same normalized target matcher. It first matches an exact period-normalized target name, then a unique Chronicle fact key, then an exact structured source/statistic/measure/dimensions identity. A key must identify one remaining target on each side; ambiguous keys are reported and left unmatched. Comparison responses include the representation of each target, the matching method, both release identifiers, and counts by matching method.

Calibration target investigations

The dashboard identifies discrepancies. Determine their cause from a target investigation packet, release artifacts, and relevant source code by following the shared investigation workflow.

node scripts/microcosm-investigation-packet.mjs \
  --release <release-id> <target-id> \
  --out investigations/latest-target-packet.json

The workflow separates reviews of Chronicle source semantics, target materialization, PolicyEngine model mapping, and calibration calculations before combining their evidence into one report.

Develop

make install   # cd frontend && bun install
make dev       # first available loopback port, starting at 3000
make typecheck # tsc --noEmit
make test      # frontend data and development-launcher tests
make build     # next build

The development launcher prints the selected dashboard URL and records its port in frontend/.next/dev-port. Set PORT to begin the search at a different port; if that port is occupied on the IPv4 or IPv6 loopback address, the launcher increments by one until it finds a port that is free on both network families. Next.js binds to 127.0.0.1, and the launcher prints that exact URL to avoid hostname resolution selecting a different local process.

Run the Python Chronicle evaluation harness and its public numerical adapter validation manually when reviewing Chronicle or dependency updates:

uv run python scripts/verify_evaluation_harness.py

The hosted variable lookup has a separate Python environment in backend/requirements.lock. Its US/Core/SPM pins protect the existing model; update them together with the certified Microcosm data release. Do not use the root evaluation lock to install this service. Check the hosted assembly without loading a population with:

uv venv --python 3.12 .tmp/hosted-runtime
uv pip install --python .tmp/hosted-runtime/bin/python \
  --require-hashes -r backend/requirements.lock
uv pip install --python .tmp/hosted-runtime/bin/python \
  'pytest>=8.3,<9' httpx==0.28.1 modal==1.5.5
uv pip check --python .tmp/hosted-runtime/bin/python
UV_PROJECT_ENVIRONMENT=.tmp/hosted-runtime uv run --no-sync \
  python frontend/scripts/verify_hosted_runtime.py
UV_PROJECT_ENVIRONMENT=.tmp/hosted-runtime uv run --no-sync \
  python -m pytest frontend/tests -q

The hosted lookup accepts only the reviewed 2024 period. It passes a one-year USMultiYearDataset to the pinned country package so that the package does not create unused later-year copies of the input population.

Production deployment runs automatically after the exact main commit passes CI. The workflow deploys a commit-specific Modal backend first, builds an unaliased Vercel production candidate against that backend, runs real national and state calculations, and promotes the frontend only after those checks pass. Vercel Git deployment is disabled for main but remains enabled for pull-request previews. See the hosted runtime procedure for required secrets, project identities, verification, and rollback.

/api/microcosm_variable?metadata=1 reports installed packages and the deployment commit, when available, without downloading a dataset. The mounted equivalent is /calibration/dashboard/api/microcosm_variable?metadata=1. Its data_configuration describes repository settings, not a verified loaded release. Calculation responses include observed runtime versions. See the hosted runtime procedure for the separate canonical model/data migration and deployment verification gates.

See the Chronicle update workflow for the optional Microcosm and ACS inputs and the separate full-artifact command. See calibration tree artifacts for the breadth-first folder schema, filtering indices, Hugging Face webhook publication, Vercel Blob authentication, one-time historical publication, and cache behavior.

The US variable-calculation service uses its reviewed immutable release. Leave POPULACE_HF_REPO and POPULACE_HF_REVISION unset for that service; conflicting overrides fail validation. Dashboard release discovery is separate: automated publication resolves Hugging Face sources to exact commits and records every immutable build in the dashboard's Vercel Blob manifest. The dashboard resolves both current and historical calibration maps through that manifest and uses a content-derived build artifact ID as the CDN path. Optional POPULACE_UK_HF_REPO, POPULACE_UK_HF_REVISION configure the UK; and POPULACE_BE_HF_REPO, POPULACE_BE_HF_REVISION for Belgium. The Belgium repository defaults in code to policyengine/populace-be-private. Set HF_TOKEN or HUGGINGFACE_TOKEN to read private datasets. Each country with the staging capability declares its own staging repository in the country registry; it never falls back to another country's repository. US staging defaults to policyengine/populace-us-staging; override it with the backward-compatible POPULACE_STAGING_HF_REPO and POPULACE_STAGING_HF_REVISION variables. UK staging defaults to the private dataset policyengine/populace-uk-staging; configure its server-only read credential as POPULACE_UK_STAGING_HF_TOKEN, and optionally override the dataset or revision with POPULACE_UK_STAGING_HF_REPO or POPULACE_UK_STAGING_HF_REVISION. Never use a NEXT_PUBLIC_ variable for a private-repository credential.

A staging registration may also list local_area_pipelines: the pipeline.id values, as run manifests record them, of pipelines whose candidates are local-area builds. The Staging page and /api/microcosm/staging/compare compare those candidates with the newest published local-area build instead of the current release, and the page names that release in its validation table and calibration map. The UK lists uk-local-candidate, Microcosm's local-area candidate pipeline; its national attempts run uk-frs-calibration. A country that lists none is unaffected.

The staging contract fixtures under frontend/packages/calibration-tree/src/fixtures/staging-contract/ are byte-identical copies of Microcosm's canonical producer fixtures. The consumer tests pin each version's SHA256SUMS digest and verify every listed file. A contract change must update the canonical Microcosm fixture, its digest manifest, this copy, and both repositories' contract tests in their respective feature branches.