VaireDB

VaireDB is a multi-instance DuckDB coordinator with builtin distributed query engine provided by Datafusion & Balista. Also combile PostgreSQL wire compatibility, with DuckDB's columnar vectorized execution engine for high-throughput analytical workloads across a horizontally scalable cluster.

Read the When use VaireDB section for understanding where VaireDB is best suited for your use cases.

[!IMPORTANT]
VaireDB is currently under active development.

Breaking changes may occur and major features may be added. Consider this a work in progress and not yet ready for production use. Any contributions are welcome for targeting the production-ready v1.0 release.

[!IMPORTANT]
AI full disclosure.

This software is developed with strong assistance from AI coding agents and with humans leading the ideas, testing, and debugging. We say this openly because it shaped how the project was built. If you are not happy with AI-developed code, this software is not for you.

Overview

VaireDB exposes a unified SQL interface through the PostgreSQL wire protocol (v3), allowing connections from any standard PostgreSQL client (psql, JDBC, etc.). Under the hood, data is hash-sharded across core nodes, each running an embedded DuckDB instance optimized for OLAP queries.

Key characteristics:

• PostgreSQL-compatible SQL via DataFusion parser with automatic dialect translation to DuckDB.
• Horizontal scalability through hash sharding across core nodes (range sharding planned).
• Analytical performance powered by DuckDB's columnar vectorized engine.
• Fault tolerance with configurable replication factor and quorum-based writes.
• Column pseudonymization for compliance: declared columns are HMAC-SHA256 hashed in the coordinator so plaintext never reaches storage.
• Operational simplicity as self-contained Rust binaries with YAML configuration.

When use VaireDB

Use Cases

• CQRS read side — an analytical database positioned close to microservices' transactional databases, serving the query (read) side of a CQRS architecture. Transactional systems keep handling writes in their own stores, while VaireDB absorbs the heavy read and aggregation traffic that would otherwise contend with operational workloads — keeping write paths fast and read paths scalable.

• Shared read model — a denormalized, read-optimized view of data shared across a microservices ecosystem. Instead of each service repeatedly joining and reshaping data from many sources, VaireDB holds a consolidated, query-friendly representation that teams can reuse, reducing duplicated effort and keeping cross-service reporting consistent.

• Data fabric or micro-fabric — deployable both as a wide data fabric spanning the whole ecosystem and as a micro-fabric local to each bounded context. The same engine scales from a single bounded context to an organization-wide layer, so you can start small within one domain and grow toward a shared analytical backbone without changing technology.

• Compliance datastore — supporting data take-out (export), deletion, and anonymization. By centralizing a queryable copy of data, VaireDB makes it easier to satisfy regulatory obligations such as subject-access exports, right-to-be-forgotten deletions, and anonymization, without hunting through every individual service store.

Architecture

VaireDB follows a coordinator/worker topology:

┌──────────────────────────────────────────────────┐
│                   Clients                         │
│            (psql, JDBC, any PG driver)           │
└──────────────────────┬───────────────────────────┘
                       │ PostgreSQL wire protocol (port 5432)
                       ▼
┌──────────────────────────────────────────────────┐
│               Coordinator Node                    │
│  ┌─────────┐ ┌────────┐ ┌──────────────────┐    │
│  │ Catalog │ │ Router │ │ Ballista Sched.  │    │
│  └─────────┘ └────────┘ └──────────────────┘    │
└──────┬───────────────┬───────────────────────────┘
       │ gRPC writes   │ reads (Ballista)
       ▼               ▼
┌────────────┐  ┌────────────┐  ┌────────────┐
│ Core Node  │  │ Core Node  │  │ Core Node  │
│  (DuckDB)  │  │  (DuckDB)  │  │  (DuckDB)  │
└────────────┘  └────────────┘  └────────────┘

• Coordinator handles client connections, SQL parsing, query planning, metadata management, and dispatching work to core nodes.
• Core Nodes store data in DuckDB, execute shard-local queries, and replicate writes.

For full details, see the Architecture Documentation.

Project Documentation

VaireDB Documentation

System Design Documentation

All documentation lives under docs/, split into architecture references, feature designs, and testing notes.

Architecture (docs/specs/):

以上为 README 的开头部分。其余内容请阅读原文。 查看完整 README ↗