Graph engine for context assembly. Object-storage native. Agents work with graph data the way engineers work with code: branched, typed, reviewed, merged, audited, and governed.
Busabase is an open-source (MIT) database and workspace for AI agents — data, docs, skills, and apps in one place. Run it locally, self-host it inside your network, or use Busabase Cloud, and turn age
Track, visualize, and compete on AI coding assistant token usage across Claude Code, Cursor, OpenCode, Codex, Gemini, Kimi, and Qwen. The Kardashev Scale for AI Devs.
8h +3★ 298Data
Site preview image
Do you operate this project?
Owners of public repositories under a personal GitHub account can sign in to verify ownership and manage the introduction themselves. No administrator approval is needed.
Organization repository · verify with a repository file
Sign in, generate a verification file, and commit it to this repository’s default branch within 30 minutes. Files on other branches or forks do not count. Then check the file below to manage the project without administrator approval. Keep the file to retain your access.
If needed, you will go to sign-in. After signing in, return here and press your chosen verification button again.
Run /nomorevibe verify in your project folder to verify ownership and manage the introduction yourself.
Current architecture/runbook: docs/CURRENT_STATE.md.
Agent-ready remediation packages: docs/fixes/README.md.
The 2026-10-03 audit found unresolved aggregation, API safety and export issues;
see the prioritized backlog before relying on derived totals or full downloads.
Totals and averages. INS's "Total" rows are not in our downloads, so every total
shown is computed. Summed counts (people, stays) equal INS's totals exactly. Averages
and rates (wages, unemployment rate) are never summed. Where shown, they are weighted
reconstructions from INS-published parts, which are rounded, so they can differ from
INS's own total by one unit in the last digit (e.g. 5,097 vs 5,098 lei). Details:
docs/METHODOLOGY.md.
The app exposes a minimal SDMX 2.1 REST API (agency INS) compatible with sdmxthon and the SDMX Dashboard Generator:
Endpoint
Description
GET /sdmx/2.1/data/INS,{flow}/{key}
GenericData XML — dot-notation key, + OR separator, startPeriod/endPeriod/lastNObservations query params
GET /sdmx/2.1/datastructure/INS/{flow}/1.0
DataStructure Definition (DSD) XML with codelists
GET /sdmx/2.1/dataflow/INS/{flow}/1.0
Dataflow definition XML
Example: /sdmx/2.1/data/INS,ACC102B requests observations for dataset ACC102B.
The current endpoint silently caps results at 50,000 observations; see
SDMX limitations and complete-export spec.
Script numbers are historical, not execution order: import metadata → classify
dimensions → build stage-11 SDMX mappings → stage-9 conversion → split/profile.
The full run order by phase is in AGENTS.md.
Fetching scripts accept --lang; other scripts have different options and/or
use TEMPO_LANG. Canonical output is shared across languages; the incremental
--lang en path is not a verified safe refresh. See CURRENT_STATE.md
and BILINGUAL.md before running writers.
Converts CSVs directly to canonical SDMX parquet — maps values via sdmx_codes, renames columns via sdmx_column_map, never writes NULL
10
10-import-metadata.py
DuckDB tables
Reconciles matrices/dimensions/options with data/2-metas (new options/periods of existing datasets included); --matrix A,B targets, --stats-only, --dry-run
10
10-classify-dimensions.py
dimension_options_parsed, matrix_profiles
Parses/classifies dimensions, detects archetypes; --matrix A,B refreshes only those (tables kept)
10
Incremental Update (update-pipeline.py)
Orchestrates incremental updates from the INS news feed. Every matrix it touches was named explicitly (--matrix), flagged by INS as changed (feed) or owed from a previous failed run (retry set), so it force-refreshes (re-fetches metadata, CSV, parquet, SDMX) by default.
python update-pipeline.py # feed entries since the watermark + retry set
python update-pipeline.py --refetch-news # re-fetch news CSV first
python update-pipeline.py --since 01.03.2026 # feed entries dated >= this (inclusive); never lowers the watermark
python update-pipeline.py --all # whole feed, ignore the watermark
python update-pipeline.py --matrix ACC101B # exactly these codes; never moves the watermark
python update-pipeline.py --skip-existing # resume/debug: skip fetch/convert if files exist (partial run)
python update-pipeline.py --dry-run # plan only: writes no state, log, DB or file
python update-pipeline.py --strict # optional-stage failures also exit 1
Per-matrix steps, in dependency order (each return code is checked; the first failed
required stage stops that matrix, the others still run):
9-csv-to-parquet → 12-split-datasets --matrix (children replaced as a set)
→ 10-import-metadata --matrix --stats-only (row_count/size/path of the new parquet)
optional: 13-dimension-structure and generate_view_profiles for the parent and its children
validate (parent/child parquet readable, row counts equal the DB, children registered)
Run level afterwards: 4-build-meta-index, date sync. Coverage, trends, value profiles
and the search index have no per-matrix mode: the touched matrices are recorded under
stale in the state file; --global-profiles runs those four whole-corpus scripts
(optional stages). Flags: --skip-duckdb leaves the DB refresh stages out (partial run).
Required (any failure: exit 1, matrix goes to the retry set, watermark frozen):
metadata, CSV (6-fetch-csv), import/classify/code maps, conversion (9), split (12,
unless --no-split), stats refresh, validation, and the batch index/date-sync.
A failed --fetch-context aborts the run.
Optional (recorded and printed, exit 0 unless --strict): 13-dimension-structure,
generate_view_profiles, --global-profiles scripts. The matrix is marked ok_degraded;
derived features are not verified.
Empty at source (6-fetch-csv exit 3) is recorded as empty: not retried, not a failure.
Nothing is deleted on failure; the previous parquet/profile artifacts stay.
State: data/logs/update-pipeline-state.json (--state-file) holds the watermark,
retry set and per-matrix/per-stage outcomes with reasons and source update dates.
last-pipeline-run.txt mirrors the watermark for legacy readers.
Watermark = newest feed date of a fully successful run (not wall-clock today). It is
inclusive, so the latest day is reprocessed next run. Partial runs (--skip-existing,
--no-split, --skip-duckdb) and --matrix runs never advance it.
Language: only ro. --lang en / TEMPO_LANG=en are rejected (exit 2) because the
processing stages write shared canonical output; children always get TEMPO_LANG=ro.
Still open under FIX-03: generation manifests,
corpus repair/quarantine. scripts/audit-corpus.py --data-dir DIR [--json-out F] is a read-only,
deterministic corpus audit (categories, schema, NULL dims, time validity, grain, coverage).
Canonicalizes parquet files in data/corpus/parquet/
chart-taxonomy.py
Generates chart-taxonomy report
cleanup-view-profiles.py
Removes orphan view-profiles
detect-totals.py
Detects "Total" rows for stripping
normalize-labels.py
Normalizes dimension option labels
prepare-deploy-data.sh
Atomically stages corpus + DuckDB + search.duckdb + view-profiles + MANIFEST.json into deploy-data/
release-check.py
Validates staged data (manifest/hashes, FTS, index generation, parquet coverage), runs tests, optional docker smoke (--local: the same smoke without Docker), --deploy only on request, --rollback
profile-values.py
Computes per-dataset value profiles → dataset_value_profiles
strip-totals-from-parquet.py
Strips precomputed "Total" rows from canonical parquets
data/
1-indexes/{lang}/ context.csv, matrices.csv
2-metas/{lang}/ {dataset-id}.json — metadata per dataset
4-datasets/{lang}/ raw CSVs from TEMPO API
4-datasets-slim-samples/ 50/ and 100/ row samples for LLM analysis
parquet-v2/ro/ Parquet v2 (numeric IDs) — dead since 2026-09-05 (stage 9 no longer
reads it); still read by 12-split-datasets.py for v2-sourced splits
corpus/ ← used by app
parquet/ SDMX-format parquet files — 4,274 on disk (2026-10-03; includes leftovers)
metadata.duckdb Main DuckDB metadata (inspect actual schema)
search.duckdb Search index DB
view-profiles/ Per-dataset JSON view profiles — 3,679 files (2026-10-03)
eval/ Eval baselines (chart_selector, agent_search)
meta/ Reference data (judet CSVs, SIRUTA)
logs/ Pipeline execution logs (incl. last-pipeline-run.txt)
sdmx-dashboards/ SDMX dashboard YAML configs
_obsolete/ Archived legacy intermediates (parquet-v3, 5-compact-datasets, 6-sdmx-csv, 3-db)
scripts/prepare-deploy-data.sh — stages metadata.duckdb, search.duckdb, parquets, view profiles and a MANIFEST.json (file sizes + sha256, source build times, latest source observation date; the generation block is a placeholder until FIX-03)
scripts/release-check.py — the release gate
Ask/MCP content logs (mcp-calls.jsonl, ask-chats.jsonl) live on the Fly /logs volume and are read with fly ssh/fly sftp; see the runbook in AGENTS.md ("Reading Ask and MCP content logs on Fly")
Release flow (staging must precede the image build; fly deploy never runs implicitly):
source ~/devbox/envs/240826/bin/activate
python scripts/build-search-index.py # if metadata changed (index must cover all canonical matrices)
bash scripts/prepare-deploy-data.sh # builds in deploy-data.tmp.*, validates, swaps atomically; previous kept as deploy-data.prev
python scripts/release-check.py --docker # manifest/hashes, FTS, tests, image build + smoke on :8095
python scripts/release-check.py --local --stage DIR # same smoke on this checkout without Docker (not an image/memory result)
python scripts/release-check.py --docker --deploy # same gates, then `fly deploy`
python scripts/release-check.py --rollback # swap deploy-data <-> deploy-data.prev (then rebuild/redeploy the image)
A failed staging leaves the previous deploy-data/ untouched. The image installs the DuckDB fts extension at build time and fails the build if search.duckdb can't be opened. GET /api/health reports search.mode (fts or fallback; production fallback to name matching is also logged at ERROR) and the staged generation/observation dates.
Clean checkout (no corpus, no secrets): pip install -r requirements-dev.txt && python -m pytest tests -q. Tests marked corpus are skipped without data/corpus/; a skip is not a pass. Dependency sets: requirements.txt (runtime/image), requirements-pipeline.txt, requirements-dev.txt. CI: .github/workflows/ci.yml (synthetic tests only; the local fly-deploy.yml stays untracked).
Natural-language data agent via validated service tools; no generated SQL
Ask budgets/privacy and shared safe aggregation (FIX-08/FIX-02)
Notebook-ready exports, publish to Kaggle
Basic stats/charts per localități (normalize to population)
Static site migration (DuckDB-WASM, Cloudflare Pages)
Source API notes
Atentie! Nomenclatoarele care prezinta doar optiunea "Total" se vor completa automat cu alte optiuni doar daca nomenclatorului anterior i se deselecteaza optiunea "Total" si i se alege o singura alta optiune,
Important! Unitatile de masura sunt implicit selectate toate pentru a preveni rezultate goale atunci cand se combina cereri incompatibile (spre exemplu, s-ar putea selecta productia de porumb in litri; sau o valoare in ROL dupa denominarea din 2005, etc.).
docs/ Summary
Core Architecture
METHODOLOGY.md — How computed totals, averages and rates relate to the values INS publishes.
reports/ins-notes.md — Per-dataset notes and decisions, and the points to raise with INS (generated from app/data/dataset_notes.json).
CURRENT_STATE.md — Current paths, dependency order, serving behavior and unresolved operating limits.
DUCKDB_SPECS.md — Historical schema design; actual DB introspection is authoritative.
DUCKDB_GUIDE.md — Historical query examples; use current paths and verify aggregation grain.
Builds SDMX code mappings (sdmx_codes, sdmx_column_map — stage 9 depends on these). The no-flag run drops and rebuilds both tables and is unsafe for canonical matrices; use --matrix A,B for refreshes
11
11-coverage-profiler.py
dataset_coverage DuckDB table
Analyzes data completeness
12
12-parquet-to-sdmx.py
(deprecated, not run)
Read a dead parquet-v2/ snapshot; stage 9 writes SDMX directly now (2026-09-05) — see docs/reports/stage9-sdmx-migration.md
12
12-split-datasets.py
data/corpus/parquet/
Splits inconsistent datasets into clean sub-datasets; each parent's children are staged, validated and swapped in with their DB rows as one set (failure keeps the previous generation)
13
13-dimension-structure.py
dimension_structure DuckDB table
Verifies each dimension's levels, real aggregates, additivity and nesting
