Duckstring

Get your ducks in a row.

Duckstring is an open source data engineering platform built on DuckDB. Data into the terabytes fits comfortably on a single machine, which is where DuckDB is at its best, and Duckstring provides what a pipeline needs around it: packaging and versioning, orchestration, incremental processing, a catalog and delivery. It runs the same on a laptop, on a single server, or with work spread across cloud machines.

Documentation · Quickstart · Playground

Versioned transformations

Each set of transformations is a Pond: a Python project whose pond.toml names it, gives it a semantic version and lists the Ponds it reads from.

[pond]
name = "sales"
version = "1.2.0"

[sources]
transactions = "1.0.0"
products = "1.1.0"

The pipeline follows from those declarations, so it never has to be drawn by hand. A breaking change ships as a new major version, which runs alongside the old one while downstream Ponds move across at their own pace.

Pull orchestration

Schedules are set on the Ponds whose output is actually used, at the end of the pipeline. Duckstring works back from there, running each upstream Pond only as often as something downstream needs it, so a path nobody consumes never runs. A daily job is one command:

duckstring refresh add reports --every 1d

Data can also stay fresh just because it's being used: a Tap gives an app its own key, and its queries refresh the Pond when the data is older than you allow, so nothing runs while the app is idle.

The Playground runs the demo pipeline in your browser, with a short guided tour of how it's scheduled.

Incremental processing

A Trickle stores a table's changes as a Z-set: rows weighted +1 when added and -1 when removed. Joins and aggregations written with the Trickle builder recompute only the keys that changed since the last run, and handle updates and deletes correctly across any shape of join:

from duckstring import ripple


@ripple
def priced_line(pond):
    (
        pond.trickle("orders.order_line")
        .join(pond.trickle("catalog.product"), on="product_id")
        .select(
            "s0.order_id, s0.product_id, s0.quantity, s1.unit_price, "
            "CAST(round(s0.quantity * s1.unit_price, 2) AS DECIMAL(14,2)) AS revenue"
        )
        .merge("priced_line", pk="order_id")
    )

Code and data together

The runtime, called the Catchment, is also the catalog. Each major version of a Pond is a schema holding its tables, with lineage, run history and the version's schema contract recorded alongside. Query it from the CLI, from Python, or from any Postgres client:

SELECT * FROM reports_v1.monthly_summary;

Python and the command line

Ripples, the steps inside a Pond, are Python functions with a DuckDB connection, so SQL, DuckDB's relation API, pandas and Arrow all work. Everything a Catchment does can be driven from the duckstring CLI or its HTTP API, and a web UI shows pipelines as they run.

Quickstart

Duckstring needs Python 3.10 or newer.

pip install duckstring
duckstring catchment init --name dev

This starts a local Catchment, with its web UI at http://127.0.0.1:7474. Leave it running, and in a second terminal:

mkdir demo && cd demo
duckstring pond demo                       # transactions, products → sales → reports
duckstring pond deploy --all --yes
duckstring refresh now reports                 # run the pipeline once, from the end
duckstring query reports.monthly_summary

The Quickstart goes through the same steps with explanations.

Also included

• Local testing against small samples of each Source, called Puddles
• dbt projects deployed as Ponds, one step per model
• Delivery to Postgres (incremental and exactly once), S3, Google Cloud Storage and local files
• Retries, alerts and Prometheus metrics
• Per-Pond Python environments from pyproject.toml and uv.lock
• Cloud compute on AWS Fargate or EC2, chosen per Pond, with data on S3
• Connected Catchments, where a Pond reads from a Pond on another Catchment

Documentation

docs.duckstring.com has the concepts, task guides, and a reference for the CLI, Python API, pond.toml and HTTP API. To try pull orchestration without installing anything, the Playground simulates it in the browser, with a guided tour.

Questions and discussion

Ask questions, share what you're building or suggest ideas in GitHub Discussions. Bugs go in Issues.

Status

Duckstring is pre-1.0, so interfaces may still change between minor versions. Changes are listed in the changelog.

Hosted Catchments

A hosted Catchment service is planned at duckstring.com. If you're interested, get in touch at [email protected].

License

Apache 2.0