ins.gov2.ro — descarcă, normalizează + UI/navigator pentru datele oferite de Institutul național de statistică, via TEMPO-Online (statistici.insse.ro)

Folosit și de geo-spatial.org Data Hub

INS Tempo Online but make it nice.

📋 Backlog · 📅 Activity History · 🤖 LLMs.txt

Current architecture/runbook: docs/CURRENT_STATE.md. Agent-ready remediation packages: docs/fixes/README.md. The 2026-10-03 audit found unresolved aggregation, API safety and export issues; see the prioritized backlog before relying on derived totals or full downloads.

Totals and averages. INS's "Total" rows are not in our downloads, so every total shown is computed. Summed counts (people, stays) equal INS's totals exactly. Averages and rates (wages, unemployment rate) are never summed. Where shown, they are weighted reconstructions from INS-published parts, which are rounded, so they can differ from INS's own total by one unit in the last digit (e.g. 5,097 vs 5,098 lei). Details: docs/METHODOLOGY.md.

---

Running

What	Command	URL
Main app (FastAPI + DuckDB)	uvicorn app.main:app --reload --port 8080	http://localhost:8080
DuckDB browser	python duckdb-browser.py	http://localhost:5000

Activate venv first: source ~/devbox/envs/240826/bin/activate

Web Application (app/)

FastAPI backend with DuckDB + Parquet, Vanilla JS + ECharts frontend.

app/
  main.py             — FastAPI entry, mounts API routers + static files
  config.py           — DB_PATH, PARQUET_DIR (corpus/parquet), MAX_DATA_ROWS=50000
  db.py               — DuckDB cursor-per-request (concurrency-safe)
  routers/            — /api/categories, /api/datasets, /api/datasets/{id}/data, /api/datasets/{id}/download, /sdmx/, /api/ask, /places

SDMX 2.1 REST API

The app exposes a minimal SDMX 2.1 REST API (agency INS) compatible with sdmxthon and the SDMX Dashboard Generator:

Endpoint	Description
GET /sdmx/2.1/data/INS,{flow}/{key}	GenericData XML — dot-notation key, + OR separator, startPeriod/endPeriod/lastNObservations query params
GET /sdmx/2.1/datastructure/INS/{flow}/1.0	DataStructure Definition (DSD) XML with codelists
GET /sdmx/2.1/dataflow/INS/{flow}/1.0	Dataflow definition XML

Example: /sdmx/2.1/data/INS,ACC102B requests observations for dataset ACC102B. The current endpoint silently caps results at 50,000 observations; see SDMX limitations and complete-export spec.

  services/           — dataset_search, dataset_meta, query_builder, chart_selector, dashboard_composer, insights,
                        headlines, dimension_structure, place_service, agent, llm_client
  static/             — dataset-v2.html + js/dashboard-v2.js (main dataset page), index.html + js/explore-app.js (v1
                        explorer), places/place, compare, ask, dimensions-explorer pages
  static/js/          — chart-factory → chart-geo, chart-demographic, chart-new-types; site-chrome, filter-panel, …
  static/geo/         — romania-counties/regions/macroregions.geojson

Pipeline Scripts

Script numbers are historical, not execution order: import metadata → classify dimensions → build stage-11 SDMX mappings → stage-9 conversion → split/profile. The full run order by phase is in AGENTS.md. Fetching scripts accept --lang; other scripts have different options and/or use TEMPO_LANG. Canonical output is shared across languages; the incremental --lang en path is not a verified safe refresh. See CURRENT_STATE.md and BILINGUAL.md before running writers.

#	Script	Output	Description
1	1-fetch-context.py	data/1-indexes/{lang}/context.csv	Fetches category hierarchy from TEMPO API
2	2-fetch-matrices.py	data/1-indexes/{lang}/matrices.csv	Fetches dataset list
3	3-fetch-metas.py	data/2-metas/{lang}/{id}.json	Downloads metadata per dataset
4	4-build-meta-index.py	data/1-indexes/{lang}/matrices-list.csv	Builds summary index from metadata
5	5-varstats-db.py	data/3-db/{lang}/tempo-indexes.db	(legacy) Creates SQLite DB from metadata
6	6-fetch-csv.py	data/4-datasets/{lang}/	Downloads raw CSV data from TEMPO API
7	7-data-compactor.py	data/5-compact-datasets/{lang}/	(legacy) Replaces text labels with numeric IDs
8	8-setup-duckdb-schema.py	data/corpus/metadata.duckdb	Creates DuckDB schema (contexts, matrices, dimensions)
9	9-csv-to-parquet.py	data/corpus/parquet/	Converts CSVs directly to canonical SDMX parquet — maps values via sdmx_codes, renames columns via sdmx_column_map, never writes NULL
10	10-import-metadata.py	DuckDB tables	Reconciles matrices/dimensions/options with data/2-metas (new options/periods of existing datasets included); --matrix A,B targets, --stats-only, --dry-run
10	10-classify-dimensions.py	dimension_options_parsed, matrix_profiles	Parses/classifies dimensions, detects archetypes; --matrix A,B refreshes only those (tables kept)
10

Incremental Update (update-pipeline.py)

Orchestrates incremental updates from the INS news feed. Every matrix it touches was named explicitly (--matrix), flagged by INS as changed (feed) or owed from a previous failed run (retry set), so it force-refreshes (re-fetches metadata, CSV, parquet, SDMX) by default.

python update-pipeline.py                        # feed entries since the watermark + retry set
python update-pipeline.py --refetch-news         # re-fetch news CSV first
python update-pipeline.py --since 01.03.2026     # feed entries dated >= this (inclusive); never lowers the watermark
python update-pipeline.py --all                  # whole feed, ignore the watermark
python update-pipeline.py --matrix ACC101B       # exactly these codes; never moves the watermark
python update-pipeline.py --skip-existing        # resume/debug: skip fetch/convert if files exist (partial run)
python update-pipeline.py --dry-run              # plan only: writes no state, log, DB or file
python update-pipeline.py --strict               # optional-stage failures also exit 1

Per-matrix steps, in dependency order (each return code is checked; the first failed required stage stops that matrix, the others still run):

1. metadata fetch → 6-fetch-csv
2. 10-import-metadata --matrix (changed options/periods) → 10-classify-dimensions --matrix → 11-build-sdmx-codes --matrix (stage 9 resolves labels/columns through these)
3. 9-csv-to-parquet → 12-split-datasets --matrix (children replaced as a set) → 10-import-metadata --matrix --stats-only (row_count/size/path of the new parquet)
4. optional: 13-dimension-structure and generate_view_profiles for the parent and its children
5. validate (parent/child parquet readable, row counts equal the DB, children registered)

Run level afterwards: 4-build-meta-index, date sync. Coverage, trends, value profiles and the search index have no per-matrix mode: the touched matrices are recorded under stale in the state file; --global-profiles runs those four whole-corpus scripts (optional stages). Flags: --skip-duckdb leaves the DB refresh stages out (partial run).

• Required (any failure: exit 1, matrix goes to the retry set, watermark frozen): metadata, CSV (6-fetch-csv), import/classify/code maps, conversion (9), split (12, unless --no-split), stats refresh, validation, and the batch index/date-sync. A failed --fetch-context aborts the run.
• Optional (recorded and printed, exit 0 unless --strict): 13-dimension-structure, generate_view_profiles, --global-profiles scripts. The matrix is marked ok_degraded; derived features are not verified.
• Empty at source (6-fetch-csv exit 3) is recorded as empty: not retried, not a failure.
• Nothing is deleted on failure; the previous parquet/profile artifacts stay.
• State: data/logs/update-pipeline-state.json (--state-file) holds the watermark, retry set and per-matrix/per-stage outcomes with reasons and source update dates. last-pipeline-run.txt mirrors the watermark for legacy readers.
• Watermark = newest feed date of a fully successful run (not wall-clock today). It is inclusive, so the latest day is reprocessed next run. Partial runs (--skip-existing, --no-split, --skip-duckdb) and --matrix runs never advance it.
• Language: only ro. --lang en / TEMPO_LANG=en are rejected (exit 2) because the processing stages write shared canonical output; children always get TEMPO_LANG=ro.
• Exit codes: 0 ok, 1 required failure, 2 usage / unsafe mode.

Still open under FIX-03: generation manifests, corpus repair/quarantine. scripts/audit-corpus.py --data-dir DIR [--json-out F] is a read-only, deterministic corpus audit (categories, schema, NULL dims, time validity, grain, coverage).

Other root-level scripts

Script	Description
generate_view_profiles.py	Generates per-dataset JSON view profiles → data/corpus/view-profiles/
generate_sdmx_yaml.py	Generates SDMX dashboard YAML configs → data/sdmx-dashboards/
build-geo-regions.py	Dissolves county GeoJSON into regions + macroregions
scripts/build-static.py	Builds the static site (shared hosting) from a generation; see docs/DEPLOY-STATIC.md
split_rules.py	Split rules engine — classifies datasets needing structural splits
sdmx_labels.py	Shared norm_label/parse_time_period used by stages 9 and 11 so their label matching agrees
detect_trends.py	Detects trends, YoY growth, seasonality → dataset_trends DuckDB table
duckdb_config.py	Central config: paths for all DuckDB/Parquet processing
duckdb-browser.py	Flask browser for exploring DuckDB + Parquet data
get-news.py	Scrapes INS news/press releases → data/insse_news.csv
test_chart_selector.py	Reports chart-selection results across all datasets (a report, not a test — tests live in tests/)

scripts/

Script	Description
audit-corpus.py	Audits the canonical corpus (parquet + DuckDB + view-profiles)
build_chart_selector_baseline.py	Builds eval baseline → data/eval/chart_selector_baseline.json
build_agent_search_baseline.py	Builds eval baseline → data/eval/agent_search_baseline.json
build-search-index.py	Builds search index → data/corpus/search.duckdb
build-i18n-dictionary.py	Builds bilingual labels dictionary → labels_i18n DuckDB table
canonicalize-corpus.py	Canonicalizes parquet files in data/corpus/parquet/
chart-taxonomy.py	Generates chart-taxonomy report
cleanup-view-profiles.py	Removes orphan view-profiles
detect-totals.py	Detects "Total" rows for stripping
normalize-labels.py	Normalizes dimension option labels
prepare-deploy-data.sh	Atomically stages corpus + DuckDB + search.duckdb + view-profiles + MANIFEST.json into deploy-data/
release-check.py	Validates staged data (manifest/hashes, FTS, index generation, parquet coverage), runs tests, optional docker smoke (--local: the same smoke without Docker), --deploy only on request, --rollback
profile-values.py	Computes per-dataset value profiles → dataset_value_profiles
strip-totals-from-parquet.py	Strips precomputed "Total" rows from canonical parquets

scripts/utils/

Misc one-off + legacy helpers (label conversion, slim samples, dimension index, header inventories, ad-hoc DuckDB queries).

scripts/utils/profiling/

CSV profiler + validation framework (data_profiler.py, variable_classifier.py, unit_classifier.py, validation_rules.py, build_indexes.py). See scripts/utils/profiling/readme.md.

Data

data/
  1-indexes/{lang}/        context.csv, matrices.csv
  2-metas/{lang}/          {dataset-id}.json — metadata per dataset
  4-datasets/{lang}/       raw CSVs from TEMPO API
  4-datasets-slim-samples/ 50/ and 100/ row samples for LLM analysis
  parquet-v2/ro/           Parquet v2 (numeric IDs) — dead since 2026-09-05 (stage 9 no longer
                           reads it); still read by 12-split-datasets.py for v2-sourced splits
  corpus/                  ← used by app
    parquet/               SDMX-format parquet files — 4,274 on disk (2026-10-03; includes leftovers)
    metadata.duckdb        Main DuckDB metadata (inspect actual schema)
    search.duckdb          Search index DB
    view-profiles/         Per-dataset JSON view profiles — 3,679 files (2026-10-03)
  eval/                    Eval baselines (chart_selector, agent_search)
  meta/                    Reference data (judet CSVs, SIRUTA)
  logs/                    Pipeline execution logs (incl. last-pipeline-run.txt)
  sdmx-dashboards/         SDMX dashboard YAML configs
  _obsolete/               Archived legacy intermediates (parquet-v3, 5-compact-datasets, 6-sdmx-csv, 3-db)

Deployment

• Dockerfile + fly.toml — Fly.io deployment (shared-cpu-1x, 512MB, Amsterdam region)
• scripts/prepare-deploy-data.sh — stages metadata.duckdb, search.duckdb, parquets, view profiles and a MANIFEST.json (file sizes + sha256, source build times, latest source observation date; the generation block is a placeholder until FIX-03)
• scripts/release-check.py — the release gate
• Ask/MCP content logs (mcp-calls.jsonl, ask-chats.jsonl) live on the Fly /logs volume and are read with fly ssh/fly sftp; see the runbook in AGENTS.md ("Reading Ask and MCP content logs on Fly")

Release flow (staging must precede the image build; fly deploy never runs implicitly):

source ~/devbox/envs/240826/bin/activate
python scripts/build-search-index.py                 # if metadata changed (index must cover all canonical matrices)
bash scripts/prepare-deploy-data.sh                  # builds in deploy-data.tmp.*, validates, swaps atomically; previous kept as deploy-data.prev
python scripts/release-check.py --docker             # manifest/hashes, FTS, tests, image build + smoke on :8095
python scripts/release-check.py --local --stage DIR  # same smoke on this checkout without Docker (not an image/memory result)
python scripts/release-check.py --docker --deploy    # same gates, then `fly deploy`
python scripts/release-check.py --rollback           # swap deploy-data <-> deploy-data.prev (then rebuild/redeploy the image)

A failed staging leaves the previous deploy-data/ untouched. The image installs the DuckDB fts extension at build time and fails the build if search.duckdb can't be opened. GET /api/health reports search.mode (fts or fallback; production fallback to name matching is also logged at ERROR) and the staged generation/observation dates.

Clean checkout (no corpus, no secrets): pip install -r requirements-dev.txt && python -m pytest tests -q. Tests marked corpus are skipped without data/corpus/; a skip is not a pass. Dependency sets: requirements.txt (runtime/image), requirements-pipeline.txt, requirements-dev.txt. CI: .github/workflows/ci.yml (synthetic tests only; the local fly-deploy.yml stays untracked).

• scripts/deploy/oracle/ — Oracle Cloud deployment (systemd + nginx)
• scripts/deploy/hf-spaces/ — Hugging Face Spaces deployment

Roadmap

Done

• Fetch index, download CSVs, compact data
• Import into DuckDB + Parquet (v1 → v2 → v3)
• Fetch english labels (bilingual support)
• FastAPI backend (app/) with DuckDB + Parquet
• Chart framework (archetypes: geo_time, demographic, time_residence, time_series)
• Choropleth map, demographic grouped bar, line charts
• Dataset splitting (by county, age groups, multi-UM)
• View-profile generator (current coverage is incomplete; see FIX-03)
• SDMX code mappings + parquet-v3 conversion
• Deployment setup (Fly.io, Oracle, HF Spaces)
• Define charting rules + JSON chart profiles per dataset
• Filter datasets by dimension, context info, permalinks
• 30k cells limit handling in fetch

Current

• UI polish — responsive layout, chart label truncation
• URL state for core filters/chart choices and v2 tile transforms; remaining controls tracked in backlog
• Monthly → yearly aggregation toggle

Later

• SDMX generic UI framework (multi-source: Eurostat, OECD)
• Natural-language data agent via validated service tools; no generated SQL
• Ask budgets/privacy and shared safe aggregation (FIX-08/FIX-02)
• Notebook-ready exports, publish to Kaggle
• Basic stats/charts per localități (normalize to population)
• Static site migration (DuckDB-WASM, Cloudflare Pages)

Source API notes

Atentie! Nomenclatoarele care prezinta doar optiunea "Total" se vor completa automat cu alte optiuni doar daca nomenclatorului anterior i se deselecteaza optiunea "Total" si i se alege o singura alta optiune,

Important! Unitatile de masura sunt implicit selectate toate pentru a preveni rezultate goale atunci cand se combina cereri incompatibile (spre exemplu, s-ar putea selecta productia de porumb in litri; sau o valoare in ROL dupa denominarea din 2005, etc.).

---

docs/ Summary

Core Architecture

• METHODOLOGY.md — How computed totals, averages and rates relate to the values INS publishes.
• reports/ins-notes.md — Per-dataset notes and decisions, and the points to raise with INS (generated from app/data/dataset_notes.json).
• CURRENT_STATE.md — Current paths, dependency order, serving behavior and unresolved operating limits.
• DUCKDB_SPECS.md — Historical schema design; actual DB introspection is authoritative.
• DUCKDB_GUIDE.md — Historical query examples; use current paths and verify aggregation grain.
• classify-dimensions.md — Spec for dimension classification/normalization: semantic types, parsing rules, archetype detection.

Application Specs

• app-spec.md — Full v1 spec: FastAPI + DuckDB backend, ECharts frontend, all phases from data prep to UI polish.
• app-spec-v2.md — v2 redesign spec. Discovery-first approach leveraging enriched metadata (tags, trends, relationships, chart recommendations).
• chart-framework-spec.md — Generic chart selection engine: 15 chart types, dimension role assignment, filter system.

Data & Profiling

• data analysis.md — Framework for systematic profiling, classification, and dashboard generation.
• data notes.md — Raw notes on data quirks, dimension patterns, edge cases.
• TODO_COMPACTION.md — Known issues with CSV compaction and label normalization fixes.
• PROFILING_AND_EXPLORER.md — Guide to the CSV profiler and Explorer UI: output paths, validation flags, API endpoints.

UI / Legacy

• DEPLOY-STATIC.md — The static site: build, parity-proven browser engine, gate, Apache hosting.
• STATIC-CONVERSION-SUMMARY.md — Superseded earlier static scaffold (historical).
• STATIC-VS-PYTHON-COMPARISON.md — Test results showing 100% feature parity between Python and static versions.
• JUDET-SPLIT-IMPLEMENTATION.md — How large datasets are split by county (judet) to stay under the 30k-cell API limit.

Deployment

• DEPLOY-DREAMHOST.md — Deployment guide for shared hosting via Passenger WSGI.

Agents Pipeline (docs/agents/)

• README.md — Overview of the 6-agent enrichment pipeline (produces value_profiles, coverage, trends, tags, relationships, chart_recs).
• pipeline.md — Orchestration: 3 phases, 7 agents, execution order and verification steps.
• phase1-value-profiler.md — Agent 1A: min/max/mean/percentiles/distribution per dataset.
• phase1-coverage-profiler.md — Agent 1B: time/geo coverage, fill rate, sparsity.
• phase1-trend-detector.md — Agent 1C: trend direction, YoY growth, seasonality, breakpoints.
• phase2-topic-tagger.md — Agent 2A: bilingual semantic tags from context + dataset names.
• phase2-dimension-overlap.md — Agent 2B: related-dataset discovery via dimension fingerprints.
• phase2-chart-recommender.md — Agent 2C: data-driven chart type recommendations.
• phase3-ia-designer.md — Agent 3A: generates the full information architecture spec from enriched metadata.

---

Obsolete - archived

StatExplorer (explorer/)

Alternative Tableau-inspired explorer with i18n and component-based JS architecture.

explorer/
  main.py             — FastAPI ("StatExplorer")
  services/           — chart_selector, query_builder, translations
  static/js/charts/   — bar, geo, line, heatmap, pyramid, bubble, small-multiples, table
  static/js/components/ — ChartCanvas, DatasetPicker, FilterBar, LeftSidebar, TopNav
  static/js/lib/      — api, i18n, utils

Moved to docs/misc-ideas/explorer

---

Static site exploration

see docs/misc-ideas/static-site, STATIC-CONVERSION-SUMMARY.md, STATIC-VS-PYTHON-COMPARISON.md

---

Copyright (c) 2026 gov2.ro Date reutilizabile conform EUPL v1.2 (the "Licence")