clink

clink is a stream processing engine that stays correct when processes die. You write a pipeline in SQL or typed C++; clink keeps its event-time windows and keyed state, checkpoints them, and delivers exactly once into Kafka, PostgreSQL, Iceberg and Parquet on S3, GCS or Azure. Kill a Worker mid-stream and the job resumes from its last checkpoint without losing or double-counting a record.

Pipelines read from and write to more than twenty systems: brokers (Kafka, Pulsar, NATS, MQTT, RabbitMQ), change data capture from PostgreSQL, databases (ClickHouse, MySQL, Cassandra, MongoDB, Redis), lakehouse tables (Iceberg, Delta Lake), object storage and Neo4j graphs. Each sink is held to the guarantee its table declares, and a job whose pipeline cannot give it is refused at submission. Beyond windows and joins, the SQL has MATCH_RECOGNIZE for patterns across events, and ML_PREDICT and VECTOR_SEARCH to call a model or search an index from inside a query.

The same pipeline runs unchanged on a single machine, inside a service or across a cluster. clink run pipeline.sql runs it in one process, with a first result in about 155 ms, and a service embeds the engine through a C ABI or from Python. Submitted to a Coordinator, the same file runs across its Workers with failover, live rescale and upgrades through savepoints. The cluster scales out by adding Workers. On Kubernetes, a Helm chart or an operator runs it on a secured, highly available cluster. Prometheus metrics, OpenTelemetry traces, OpenLineage events and an operations console ship with it.

The correctness claim is one you can check rather than take on trust:

• Qualified under faults. The Kafka exactly-once campaign ran two hours of Worker and Coordinator kills inside commit windows, broker outages and network partitions, and all 755 windows came out byte-exact against an independent oracle. Eleven campaigns are published, each only once green (Qualification).
• A checked protocol. The exactly-once protocol is a TLA+ specification model-checked on every push, every defect the campaigns found is a mutant it must refute, and the protocol traces of real test runs are validated against it (Exactly-once specification). What each check has found, and the duplicate all three missed, is in Three checks on exactly-once.
• Measured cost. Across the 17-query nexmark suite on a five-node cluster, clink processes an event for 1.9x to 5.3x less CPU (median 2.45x) than a JVM stream processor producing identical, correctness-gated output (Benchmarks, cost and footprint).

State is also an open dataset: checkpoints are documented Arrow IPC that open in pyarrow, DuckDB or Polars, and a running job's state answers point lookups and scans over HTTP. A flight recorder can replay an incident deterministically and freeze it into a regression test.

clink is young and pre-1.0, written by one maintainer with extensive AI assistance. The specification, the qualification campaigns and the sanitizer matrix in CI are how its correctness is checked (Status and maturity).

clink is heavily inspired by Apache Flink. Flink's model of typed operator DAGs, event-time processing, in-band watermarks and checkpoint barriers, keyed state, and exactly-once semantics is the conceptual foundation this engine builds on. clink reworks that model in modern C++, around Arrow-native columnar execution.

Try it in one command

Nothing to clone, install or configure:

docker run --rm ghcr.io/orhaugh/clink-runtime:latest run /opt/clink/examples/sql/hello.sql

The pipeline and the data it reads both ship inside the image (amd64 and arm64). It groups a small stream of device readings by region and prints the result:

{"avg_reading":21.4,"peak":21.4,"readings":1,"region":"emea"}
{"avg_reading":22.65,"peak":23.9,"readings":2,"region":"emea"}
{"avg_reading":19.2,"peak":19.2,"readings":1,"region":"amer"}
...

Those rows are the point. A batch engine prints one line per region at the end; clink prints a line per update, because the aggregate is a stream and you are watching it change. Run your own file the same way by mounting it:

docker run --rm -v "$PWD:/work" ghcr.io/orhaugh/clink-runtime:latest run /work/mine.sql

To watch a cluster instead, start the quickstart: a Coordinator, two Workers and two jobs, with the ops console on the Coordinator's port.

curl -fsSLO https://raw.githubusercontent.com/orhaugh/clink/main/examples/quickstart/compose.yml
docker compose -f compose.yml up        # then open http://localhost:8081

The console ships inside every clink release (the runtime image and the SDK), and a Coordinator serves it beside its JSON API. Its source is clink-fe; the quickstart is in examples/quickstart.

Want to build something real? Follow the Kafka to clink to ClickHouse tutorial: one docker compose up gives you a broker, a database and a small clink cluster running an event-time windowed pipeline; you then kill the Worker mid-stream, watch it recover from its checkpoint, verify every result independently, and open the job's state as an Arrow table. The example lives in examples/kafka-to-clickhouse.

Build from source

clink builds against a pinned Apache Arrow toolchain in ~/.clink-deps. The bootstrap step downloads a prebuilt, checksum-pinned archive when one exists for your platform (macOS arm64, Linux x86_64/arm64; about a minute) and compiles it from source otherwise:

git clone https://github.com/orhaugh/clink && cd clink
scripts/build-arrow.sh && scripts/build-iceberg-cpp.sh   # one-time, cached
cmake -S . -B build && cmake --build build --parallel 10
ctest --test-dir build --parallel 8                      # optional

Then run a first pipeline - one process, no daemons:

printf '{"usr":"alice","amount":12}\n{"usr":"bob","amount":7}\n{"usr":"alice","amount":5}\n' > /tmp/orders.ndjson

./build/clink run -e "CREATE TABLE orders (usr VARCHAR, amount BIGINT) \
      WITH (connector='file', format='json', path='/tmp/orders.ndjson'); \
    SELECT usr, SUM(amount) AS total FROM orders GROUP BY usr"

Supported platforms: macOS (Apple Silicon is the primary development platform) and Linux (Debian-family, exercised in CI). Windows is not supported. ./build_and_test.sh is the reproducible CI-matching path, including the sanitizer matrix; CONTRIBUTING.md has the details.

One engine, two execution models

Embedded. clink run pipeline.sql runs the whole engine in one process: SQL frontend, operators, state backends, checkpointing, connectors. First result in about 155 ms from process start, a figure gated by a Release-build test so it cannot silently regress. The same engine embeds behind a pure-C ABI (libclink), from Python (pyclink), and over Arrow Flight SQL for any ADBC/JDBC client.

Distributed. The same SQL file or compiled job plugin submits to a Coordinator/Worker cluster: parallel subtasks, hash-partitioned keyed shuffles, exactly-once checkpoints, failover from the last completed checkpoint, hot per-operator rescale, and rolling upgrades through savepoints. A Helm chart and Kubernetes operator ship in-tree. SQL planning keeps keyed stages partitioned and stages that need a global view at one instance, so raising job parallelism does not turn a global result into one partial result per subtask.

clink_node --role=coordinator --port=6123 --http-port=8081 &
clink_node --role=worker --coordinator-host=127.0.0.1 --coordinator-port=6123 &
clink run pipeline.sql --coordinator-host=127.0.0.1 --coordinator-port=8081

Workers join the coordinator on its RPC port; a SQL submission goes to its HTTP port, and a compiled job plugin submits over RPC (clink run --job=pipeline.so --coordinator-host=127.0.0.1 --coordinator-port=6123).

Typed C++ pipelines (the fluent Pipeline / DataStream<T> API, keyed process functions, windows, joins, CEP) are documented with runnable programs under docs/consumer-examples/.

Showcases

Seven open-source applications built on clink, each in a real domain, each run on one machine against real systems in Docker, and each checked the same way: a deterministic tape, failures injected on purpose, and an independent checker that works out from the tape alone what every output must hold.

Showcase	The application	What it shows
market-pulse	Market data: price bars, spreads, patterns, a mistyped print	Eight SQL scenes in one process, Python, the same SQL on a cluster; state read after the job stops; incidents replayed byte for byte
card-sentry	Card fraud detection and response	C++ pattern detectors; alerts exactly once into PostgreSQL and Kafka through a Worker kill; the run checked against the exactly-once specification
grounded	Retrieval for an operations assistant	ML_PREDICT and filtered VECTOR_SEARCH in SQL, a live index, every retrieval replayable
transfer-sentry	Fraud detection on a transaction graph in Neo4j	Seven SQL jobs keep the graph, check each event against it and decide every transfer, with no reader ever running ahead of the writer
half-time	A delivery platform on the night of a final	Nineteen jobs over fifteen systems through a Worker kill, a live rescale, a savepoint and a Coordinator failover, every store checked
coldchain	Refrigerated freight monitoring	The typed C++ API with no SQL; one graph definition in one process and on a cluster, with the same answers
playhead	Video playback analytics on Kubernetes	The operator on a secured, highly available cluster; exactly-once output to Iceberg, Kafka, GCS and Azure through an upgrade, a Worker kill and a failover; a job that scales to zero and wakes on new data

What each one does, where clink does the work and what a run proves: Showcases.

Production qualification

Benchmarks say how fast an engine is; they say nothing about whether its guarantees hold when processes die at the worst possible instant. clink runs a standing qualification programme: long campaigns on disposable multi-host rigs with faults injected into the narrowest windows of the engine's own protocols, judged by an independent oracle, published only when green and with the evidence retained. A few of the published results:

Qualified	Result
Kafka exactly-once	755/755 windows byte-exact after two hours of kills inside commit windows, coordinator SIGKILLs, broker outages and partitions
Large keyed state	29 GiB of keyed state on a disaggregated backend, every key correct under the same fault battery
Wide job graphs	147 operators as 292 subtasks, exactly once under faults, 28-second recovery from a worker kill at that width
Rolling upgrade	An engine upgrade with exactly-once continuity: 2 s savepoint, 2 s restore, every event across the boundary counted once
Semantic comparison	19 of 19 queries content-equal with an independent reference engine under pre-declared judgement classes

The full table, the rig, the method and every campaign's honesty-bounded claims: Qualification.

The protocol those campaigns exercise is also written down as a TLA+ specification and model-checked on every push, with every defect the campaigns ever found kept as a mutant the checker must refute. The model found three further interleavings the campaigns had not, fixed before any rig paid for them. The engine also records a protocol trace on request, and every trace the test suite leaves is model-checked against the specification on each push, so the code's recorded behaviour and the model are held in agreement rather than assumed to be. It proves the model and the recorded runs, not the code in general: Exactly-once specification.

Status and maturity

clink is young and pre-1.0. Its guarantees are qualified within the published bounds above, not battle-tested by years of third-party production deployments. The public surface is tiered ahead of 1.0 (Compatibility): the Stable tier of the C++ API, the C ABI and the SQL dialect is declared in tracked manifests and frozen conformance suites, and every change to it before 1.0, and every Evolving-tier change after, is called out in the CHANGELOG. Durable state is treated conservatively: snapshots carry schema versions with a migrate-at-restore path, so an upgrade does not silently invalidate checkpoints or savepoints. What the engine can do, feature by feature with caveats stated, lives in the capability catalogue.

Road to 1.0

The path to 1.0 is primarily about stability and operational evidence rather than adding another broad layer of features.

Settled, each under a design record and held by gates in CI:

• Stable public APIs. The C++, C and SQL-facing interfaces that carry compatibility guarantees across the 1.x line are declared by tier and held by conformance gates: a tracked header manifest, an append-only C symbol manifest, compile-only conformance units and a frozen SQL corpus (design record 011, Compatibility). 1.0 freezes the manifest.
• Stable extension model. A compiled job or plugin loads on any engine build whose declared extension surface matches. The contract is a tracked header manifest plus the build options, pinned Arrow version and toolchain identity that shape its layout; a refusal names the differing headers, an incompatible submit is refused before any bytes ship, and out-of-tree modules build with the packaged clink_add_job_module() (design record 010).
• Native type ergonomics. One CLINK_FIELDS declaration per type derives the byte codec, the Arrow schema and columnar batcher, distributed type registration, and a shape fingerprint that refuses a restore whose field list changed without a declared version bump (design record 009).

Still open before 1.0:

• Long-duration qualification. Complete multi-day production-style campaigns and continue expanding the measured state, scale and recovery envelope.
• Operational maturity. Incorporate real-world deployment feedback, strengthen upgrade and diagnostic tooling, and close issues found by external workloads.

1.0 will mean that these compatibility and operational contracts are ready to be relied upon. It will not mean that feature development stops.

See GitHub issues for work currently in progress.

Documentation

Everything below is published at orhaugh.github.io/clink:

• Capability catalogue: the complete shipped feature surface, with caveats.
• Diagnosing a pipeline with an agent: the diagnostic surface as MCP tools, read-only, walked through a real incident by any MCP client.
• SQL reference: the supported SQL surface, from DDL through windows, joins, MATCH_RECOGNIZE, UDFs and SQL-native ML.
• Connectors: twenty-plus sources and sinks (Kafka, Postgres incl. CDC, ClickHouse, S3/GCS/Azure Parquet, Iceberg, Delta Lake, MQTT, NATS, Pulsar, RabbitMQ, Redis, MongoDB, Cassandra, Neo4j, HTTP, Avro, WebSocket and more), each with dependencies, options and delivery semantics; Kafka speaks the Confluent Schema Registry wire format (Avro, Protobuf, JSON Schema), and a new ClickHouse table should use the typed native sink, insert_format='native'.
• Internals: every subsystem documented the way its code is structured, citing sources.
• Design decisions: why the engine is built this way, trade-offs included.
• Benchmarks and cost footprint.
• Qualification.
• Tutorial: Kafka to clink to ClickHouse on your machine, with a deliberate Worker kill and independent verification (examples/kafka-to-clickhouse).
• Kafka to Neo4j example: synthetic transfers into a graph through the Neo4j sink, with a windowed fan-in indicator enriched from reference device relationships through the Neo4j lookup, a Worker kill and an independent check of the graph.

Installing

clink ships as a regular CMake package:

cmake -S . -B build -DCLINK_BUILD_TESTS=OFF -DCLINK_BUILD_EXAMPLES=OFF \
                    -DCMAKE_INSTALL_PREFIX=/usr/local
cmake --build build --parallel 10
sudo cmake --install build

Downstream projects consume it with find_package(clink REQUIRED) and link clink::clink (or per-impl targets such as clink::kafka); docs/consumer-examples/ is a complete find_package-based project to copy from. A project that would rather build clink from source pulls it in with CMake's FetchContent at a pinned tag and links the same targets; docs/consumer-examples/fetchcontent/ is that project, and CI builds it from the commit under test. The install carries the clink CLI and the clink_node cluster daemon. Optional connectors and backends are CLINK_WITH_<NAME> CMake options (default AUTO: used when the dependency is found); the SQL frontend is CLINK_BUILD_SQL=ON by default. clink --capabilities prints what any given binary was built with.

Getting help

• Bug reports: GitHub issues.
• Questions and ideas: GitHub discussions.
• Contributing a change: CONTRIBUTING.md.
• Reporting a security issue privately: SECURITY.md.

Licence and attribution

clink is licensed under the Apache License 2.0. See LICENSE and NOTICE. You are free to use clink for any purpose, including in commercial and closed-source products. If you build on it, a link back to this repository is appreciated; for write-ups, please cite it via CITATION.cff.