中文介绍

LakeSoul

Postgres-native. Vector-native. Python-native. An open lakehouse engine — a shared catalog, ACID upserts, SQL, incremental compute, vector search and a Python SDK, in Rust, over your object storage.

Point the tools you already use at it — psql, Python, pgvector-compatible SQL — and get an incremental lakehouse underneath.

Quick Start · Architecture · Docs

Postgres-native

Point any PostgreSQL client at LakeSoul. CREATE TABLE, INSERT ... ON CONFLICT and queries run in the lakehouse engine, against object storage — not in a sidecar, and not through a CDC pipeline.

CREATE TABLE orders (
  id     bigint primary key,
  region text,
  amount numeric(20,4)
) WITH (hash_bucket_num = 8);

INSERT INTO orders VALUES (1, 'APAC', 42.00)
ON CONFLICT (id) DO UPDATE
SET region = EXCLUDED.region,
    amount = EXCLUDED.amount;

SELECT region, sum(amount) FROM orders GROUP BY 1;

The wire protocol, the extended query protocol, pg_catalog and psql \d all work. This is not a drop-in PostgreSQL: UPDATE, DELETE, COPY, RETURNING and non-READ COMMITTED isolation are refused with 0A000, and writes must be autocommit.

PostgreSQL subscriptions

The same endpoint can subscribe to a source PostgreSQL database and keep a LakeSoul CDC table in sync. Define the subscription with SQL; the endpoint validates the publication and target, takes a managed snapshot, and starts a supervised local subscriber process.

CREATE SUBSCRIPTION users_from_app
CONNECTION 'postgresql://user:pass@source:5432/app'
PUBLICATION app_pub
WITH (
  source_table     = 'public.users',
  target_table     = 'users',
  snapshot         = 'managed',
  auto_create      = true,
  schema_evolution = 'auto'
);

The source needs logical WAL, a publication, and REPLICA IDENTITY FULL; the target is a LakeSoul CDC table that can be created automatically from the source schema or pre-created; a managed snapshot requires it to be empty. The subscriber adds nullable columns and widens safe types when the source schema changes. The initial copy runs in parallel across CTID ranges (snapshot_parallelism, snapshot_memory_bytes). ALTER SUBSCRIPTION ... DISABLE/ENABLE and DROP SUBSCRIPTION control the process. A subscription can map one or more source tables; each table snapshots and commits independently. See PostgreSQL subscriptions for the full setup and SQL reference.

Incremental SQL

With incremental SQL enabled for the session, an INSERT ... SELECT issued from psql updates its target incrementally: each run reads only the commits since its cursor and writes only the changed rows. The result is identical to a full recompute.

-- The target is a keyed table carrying the CDC change column "rowKinds".
-- This switch enables incremental SQL for the session.
SET lakesoul.ivm.enabled = on;

INSERT INTO daily_revenue
SELECT date_trunc('day', ts) AS day, region, sum(amount)
FROM orders
GROUP BY 1, 2;

• SQL-level incremental, not just storage-level incremental read/write
• Replay-safe: every consumed window is recorded, so a retry skips data that is already written

Vector-native

pgvector-compatible SQL over ANN indexes that live next to the data. New rows update the index; there is no rebuild and no second system to operate.

-- documents.embedding is vector(768)
SELECT id, body, embedding <-> '[0.12, 0.98, ...]' AS distance
FROM documents
WHERE tenant_id = 'acme' AND category = 'contract'
ORDER BY embedding <-> '[0.12, 0.98, ...]'
LIMIT 10;

• IVF + RaBitQ index on object storage, maintained on write
• Structured filters and ANN in one plan
• BM25 full-text search (text_score) in the same engine

Python-native

The Python SDK reads object storage directly through the Rust IO layer. No gateway, no JDBC hop — Daft and Ray shard the read across workers, PyTorch gets a dataset, Pandas and DuckDB consume Arrow.

from lakesoul import LakeSoulCatalog

catalog = LakeSoulCatalog.from_env()
scan = catalog.scan("documents").select("id", "body")
table = scan.to_arrow_table()   # or .to_daft(), .to_ray(), .to_torch()

The same table your BI dashboard queries is the table your training job reads. Video frames, robot episodes and tensors live as blobs beside their metadata.

Quick Start

Option A — the PostgreSQL endpoint

# 1. Metadata database
docker run -d --name lakesoul-pg -p 5432:5432 \
  -e POSTGRES_USER=lakesoul_test -e POSTGRES_PASSWORD=lakesoul_test \
  -e POSTGRES_DB=lakesoul_test postgres:18
docker cp script/meta_init.sql lakesoul-pg:/
docker exec -i lakesoul-pg sh -c \
  "PGPASSWORD=lakesoul_test psql -U lakesoul_test -f meta_init.sql"

# 2. The PostgreSQL endpoint: database "default" is the LakeSoul namespace
cargo run -p postgres-lakesoul -- --port 5433 \
  --warehouse-prefix file:///tmp/lakesoul/warehouse

# 3. Connect with any PostgreSQL client
psql "postgresql://lakesoul_test:[email protected]:5433/default"

CREATE TABLE t (id bigint primary key, v text);
INSERT INTO t VALUES (1, 'hello');
SELECT * FROM t;

Option B — Python

pip install lakesoul

This is the beginning of the README. Read the rest at the original. Read the full README ↗