A Lightweight Data Stack, Demonstrated End to End

A complete data platform that runs on a laptop. Public data is loaded with dlt into a DuckLake landing zone, modelled with dbt into a single DuckDB file, extended with Polars, and published as an Evidence dashboard and a monthly data release, all orchestrated as one Dagster asset graph. No cloud account, no credentials, no bill.

📊 Live dashboard · Data catalogue · Latest data release · Quickstart · Reuse this stack · The practices, indexed · What is not done yet

509 dbt tests (469 data, 40 unit) · an enforced schema contract on every mart model · the whole pipeline run offline on every pull request · the sources fetched live weekly, and the site rebuilt from them whenever its code changes

dlt  ─▶  DuckLake  ─▶  dbt  ─▶  Polars  ─▶  Evidence
 EL     raw Parquet   stg/marts   heavy T     BI-as-code
└──────────────────── Dagster ─────────────────────┘
             one asset graph, scheduled

Why look at it

• Every layer does real work. Seven public feeds go in: emissions and energy from Our World in Data, development indicators from the World Bank, electricity prices from Eurostat, exchange rates from the ECB, weather from Open-Meteo and one retailer's invoice log, plus an EU regulation transcribed as a seed. A dashboard and a queryable warehouse come out, and no number is exported by hand.
• Two grains with opposite hard parts. For the country-year facts the hard part is coverage: which country is missing from which series, and what a join quietly drops. For the 1.04M-line transaction log it is definitions: a customer return and a stock write-off are both a negative quantity.
• The tests were measured, not assumed. When seven models were deliberately broken 38 ways, the data tests caught 5. The unit tests exist because of that gap.
• CI does not depend on anyone else's API, and neither does a deploy. Every pull request runs the whole pipeline against recorded fixtures. A weekly job runs it against the live sources, opens an issue when one changes and caches what it fetched, and the site is rebuilt from that cache unless a change could alter what is fetched.
• The warehouse is published, not just the dashboard: monthly, as DuckDB and Parquet, with attribution per source and the one column that identifies a person pseudonymised on the way out.
• A model can ask the warehouse questions, but not supply the numbers. The tools do every calculation; the model picks one and writes the prose, and just ask flags any figure in its answer that no tool printed. The same tools are served over MCP, so Claude Code or an IDE can call them too.
• Dagster wraps the layers instead of replacing them. ingest, dbt and transform still run on their own, and the asset graph comes from keys the layers already share.

Quickstart

Nothing installed? opens the repo in a browser with everything installed and just setup already run. .devcontainer/ does the same in VS Code.

Locally you need uv, which manages Python and every dependency, and just, which runs the recipes:

curl -LsSf https://astral.sh/uv/install.sh | sh   # or: brew install uv
uv tool install rust-just

git clone https://github.com/Ddscully/dlt-dbt-duckdb-evidence.git
cd dlt-dbt-duckdb-evidence

just setup      # the Python environment, and dbt's packages and manifest
just run        # ingest -> dbt build -> Polars, about a minute
just report     # build the dashboard (needs Node 18+)
just sql        # query the warehouse in the DuckDB CLI
just dagster    # the same pipeline as an asset graph, UI on :3000

just on its own lists every recipe. A built tree takes about 2.6 GB, all of it gitignored; just clean reclaims the build output, and just clean deep the node_modules too.

• Offline: just test-pipeline runs the whole pipeline against recorded fixtures in under a minute, into a throwaway warehouse.
• As a service: cp .env.example .env, set PGPASSWORD, then just compose-build && just compose-up runs everything as containers, with the lake's catalog in Postgres and its Parquet in S3-compatible storage (runbook).
• Contributing: uv run pre-commit install once; CI runs the same hooks. The test tiers are in tests/README.md.

What is in the repo

Directory	Tool	Role
ingest/	dlt	one module per publisher: incremental loads with schema inference
lake/	DuckLake	the landing zone for raw data: Parquet under a catalog, with snapshot history
dbt/	dbt + DuckDB	staging → intermediate → marts in one file, with contracts and tests
transform/	Polars	the metrics SQL models badly
agent/	MCP	the analyst's tools over the marts, a model loop that calls them, and a server for any MCP client (how)
orchestration/	Dagster	one asset graph over every layer (how)
reports/	Evidence	the dashboard, as SQL and markdown
publish/		the boundary outward: the site build, the monthly release and the history it carries forward
src/modern_data_stack/		the domain-neutral code every layer calls

The dashboard and the data

The dashboard holds six analyses, from CBAM border costs to retail customer retention, each ending in a decision somebody has to make, plus a country explorer and pages on coverage, restatements and the pipeline itself. dbt's model documentation sits beside it. docs/DASHBOARD.md describes every page and how the site is deployed.

The monthly data release is the whole warehouse as a DuckDB file and a Parquet per table, which DuckDB can query over HTTPS without downloading it. docs/PUBLISHED_DATA.md has example queries.

Reuse this stack

Almost none of the wiring is specific to emissions data. dlt-dbt-duckdb-template is this repo with the subject matter taken out and one trivial source left in, green in CI from the first commit. docs/REUSING_THIS_STACK.md covers what the template cannot: what has to be rewritten, the decisions that are expensive to change later, and the invariants that fail silently.

Documentation

Doc	What it covers
docs/PRACTICES.md	the practices this repo demonstrates, the failure each prevents, and where it is in the code
docs/FOR_REVIEWERS.md	what is unfinished, the freshness SLA, run cost, what breaks at 1000×, and what I would change
docs/WAREHOUSE.md	the sources, their grains, the landing zone and the schemas
docs/ORCHESTRATION.md	the asset graph, its jobs, backfills and freshness policies
docs/DATA_QUALITY.md	tests, contracts, groups, exposures and model versions
docs/DASHBOARD.md	each dashboard page and how the site is deployed
docs/FINANCE_AGENT.md	the analyst's tools (a revenue bridge, the semantic layer's metrics, a CBAM carbon-price scenario), asked through a local model or served to any MCP client
docs/PUBLISHED_DATA.md	the monthly release and how to query it

docs/course/ teaches the warehouse to analytics engineers through failures that pass every test, such as a one-word join edit that drops two thirds of the countries. Modules 00–04 are written. docs/STYLE_GUIDE.md has the SQL conventions, reports/README.md the Evidence layer, and AGENTS.md, the guide every coding agent reads, is a map of the repo worth reading as a human too.

What this project has and has not done yet

• The engineering is built and measured: ingestion, modelling, contracts, tests, lineage, orchestration and the publication boundary.
• The analysis has not had the same scrutiny. The pipeline does what it says, but whether it asks the right questions is untested, so treat the conclusions as illustrative.
• fct_example_scope2_emissions is fabricated: twelve invented sites, shipped in the public release so the Scope 2 model has something to work on.
• Coverage thins unevenly across columns, so cross-source comparisons are often made over different sets of countries.

docs/FOR_REVIEWERS.md explains each of these, and scores the warehouse against a governance rubric it did not write, including the part where it fails.

License

Code is MIT. The data belongs to its publishers:

Source	Licence
OWID's CO₂ and energy datasets	CC BY 4.0
World Bank WDI	CC BY 4.0
Eurostat	its own reuse policy
CBAM default values	EU law, reusable under Decision 2011/833/EU
ECB euro reference rates	the ECB's reuse policy
UCI's Online Retail II (Chen, D., 2019)	CC BY 4.0
Daily capital-city weather from Open-Meteo	CC BY 4.0, generated using Copernicus Climate Change Service information (ECMWF ERA5-Land and ERA5)

Each permits redistribution with attribution, and every data release ships an ATTRIBUTION.md crediting them per source; cite the publishers for the numbers. The repository stores little of it: dbt seeds transcribed from the CBAM regulation and the World Bank's historical income classifications, and small test fixtures. Everything else is fetched at run time. Two licence decisions that shaped the warehouse are in docs/PRACTICES.md.