clink
clink is a stream processing engine that stays correct when processes die.
You write a pipeline in SQL or typed C++; clink keeps its event-time windows
and keyed state, checkpoints them, and delivers exactly once into Kafka,
PostgreSQL, Iceberg and Parquet on S3, GCS or Azure. Kill a Worker mid-stream
and the job resumes from its last checkpoint without losing or
double-counting a record.
Pipelines read from and write to more than twenty systems: brokers (Kafka,
Pulsar, NATS, MQTT, RabbitMQ), change data capture from PostgreSQL, databases
(ClickHouse, MySQL, Cassandra, MongoDB, Redis), lakehouse tables (Iceberg,
Delta Lake), object storage and graphs (Neo4j, and Relate stores embedded in
the engine). Each sink is held to the guarantee its table declares, and a job
whose pipeline cannot give it is refused at submission. Beyond windows and
joins, the SQL has MATCH_RECOGNIZE
for patterns across events, and ML_PREDICT and VECTOR_SEARCH to call a
model or search an index from inside a query.
The same pipeline runs unchanged on a single machine, inside a service or
across a cluster. clink run pipeline.sql runs it in one process, with a
first result in about 155 ms, and a service embeds the engine through a C ABI
or from Python. Submitted to a Coordinator, the same file runs across its
Workers with failover, live rescale and upgrades through savepoints. The
cluster scales out by adding Workers. On Kubernetes, a Helm chart or an
operator runs it on a secured, highly available cluster. Prometheus metrics,
OpenTelemetry traces, OpenLineage events and an operations console ship with
it.
The correctness claim is one you can check rather than take on trust:
- Qualified under faults. The Kafka exactly-once campaign ran two hours
of Worker and Coordinator kills inside commit windows, broker outages and
network partitions, and all 755 windows came out byte-exact against an
independent oracle. Eleven campaigns are published, each only once green
(Qualification).
- A checked protocol. The exactly-once protocol is a TLA+ specification
model-checked on every push, every defect the campaigns found is a mutant
it must refute, and the protocol traces of real test runs are validated
against it (Exactly-once specification).
What each check has found, and the duplicate all three missed, is in
Three checks on exactly-once.
- Measured cost. Across the 17-query nexmark suite on a five-node
cluster, clink processes an event for 1.9x to 5.3x less CPU (median
2.45x) than a JVM stream processor producing identical, correctness-gated
output (Benchmarks,
cost and footprint).
State is also an open dataset: checkpoints are documented Arrow IPC that
open in pyarrow, DuckDB or Polars, a completed checkpoint of a whole
distributed job exports as one verified dataset to query and compare
offline, and a running job's state answers point lookups and scans over
HTTP. A flight recorder can replay an incident
deterministically and freeze it into a regression test, and, for a Kafka to
ClickHouse job, say whether the table holds exactly what a completed
checkpoint owed it, once or for every checkpoint of a running job. Before an
upgrade, a candidate build can be checked against a savepoint a baseline
build retained: restored through the engine's own path, its state, output and
final state compared with the baseline's and with an independent reference
(Upgrade validation).
clink is young and pre-1.0, written by one maintainer with extensive AI
assistance. The specification, the qualification campaigns and the
sanitizer matrix in CI are how its correctness is checked
(Status and maturity).
clink is heavily inspired by Apache Flink. Flink's model of typed operator
DAGs, event-time processing, in-band watermarks and checkpoint barriers, keyed
state, and exactly-once semantics is the conceptual foundation this engine
builds on. clink reworks that model in modern C++, around Arrow-native columnar
execution.
Try it in one command
Nothing to clone, install or configure:
docker run --rm ghcr.io/orhaugh/clink-runtime:latest run /opt/clink/examples/sql/hello.sql
The pipeline and the data it reads both ship inside the image (amd64 and
arm64). It groups a small stream of device readings by region and prints
the result:
{"avg_reading":21.4,"peak":21.4,"readings":1,"region":"emea"}
{"avg_reading":22.65,"peak":23.9,"readings":2,"region":"emea"}
{"avg_reading":19.2,"peak":19.2,"readings":1,"region":"amer"}
...
Those rows are the point. A batch engine prints one line per region at
the end; clink prints a line per update, because the aggregate is a
stream and you are watching it change. Run your own file the same way by
mounting it:
docker run --rm -v "$PWD:/work" ghcr.io/orhaugh/clink-runtime:latest run /work/mine.sql
To watch a cluster instead, start the quickstart: a Coordinator, two
Workers and two jobs, with the ops console on the Coordinator's port.
curl -fsSLO https://raw.githubusercontent.com/orhaugh/clink/main/examples/quickstart/compose.yml
docker compose -f compose.yml up # then open http://localhost:8081
This is the beginning of the README. Read the rest at the original. Read the full README ↗
