Mink
Mink serves Kafka topics as Iceberg tables, through a read-only Iceberg REST catalog and an S3 endpoint. The records stay in Kafka.
Documentation · Quick start · Contribute
mink/ the node: Kafka client, schemas, Postgres store, Iceberg files and metadata, metrics, and the mink CLIconsole/ the web console, embedded in the nodespecification/ the files and metadata Mink writes
Install
(cd console && npm ci && npm run build)
cargo install --path mink/mink-cli
The console is embedded in the binary, so it builds first.
Run a node
# Kafka, Schema Registry, Postgres and RustFS
docker compose -f harness/compose.yml up -d postgres kafka rustfs schema-registry createbucket
mink serve -c mink.toml
# mink.toml
[kafka]
bootstrap = ["localhost:29092"]
[storage]
endpoint = "http://localhost:9000"
credentials = { access_key_id = "mink", secret_access_key = "minkminkmink" }
[schema_registry]
type = "confluent"
url = "http://localhost:8081"
Catalog on :8181, S3 on :9100, admin API and console on :9180. Every setting has a MINK_* env equivalent.
Docker
docker build -t mink .
Running it against the compose stack, and as a cluster: Deployment.
Use it
mink table add payments -c mink.toml
mink table list -c mink.toml
mink granularity payments day -c mink.toml
-- DuckDB
INSTALL iceberg; LOAD iceberg; INSTALL httpfs; LOAD httpfs;
CREATE SECRET mink (TYPE S3, KEY_ID 'mink', SECRET 'mink', REGION 'us-east-1',
ENDPOINT '127.0.0.1:9100', URL_STYLE 'path', USE_SSL false);
ATTACH 'warehouse' AS mink (TYPE iceberg, ENDPOINT 'http://127.0.0.1:8181', AUTHORIZATION_TYPE 'none');
SELECT * FROM mink.kafka.payments;
Test
cargo test
# service-backed tests, env-gated
TEST_POSTGRES_URL=postgres://mink:mink@localhost:5432/mink \
TEST_KAFKA_BOOTSTRAP=localhost:29092 \
TEST_S3_ENDPOINT=http://localhost:9000 \
TEST_SCHEMA_REGISTRY=http://localhost:8081 \
cargo test
# end to end in Docker: DuckDB, PyIceberg, Spark, Trino and the console
scripts/e2e.sh
