ScrivaDB

A lightweight, append-only, file-based document database — single binary, zero dependencies, human-readable storage.

---

Install

# Homebrew (via the tap; published when a release has the tap token)
brew install --cask srjn45/scriva/scriva

# Go
go install github.com/srjn45/scriva/cmd/scriva@latest

# Docker (published GHCR image; explicitly expose its loopback listeners)
docker run -p 5433:5433 -p 8080:8080 -e SCRIVA_API_KEY=dev-key ghcr.io/srjn45/scriva \
  --grpc-addr 0.0.0.0:5433 --rest-addr 0.0.0.0:8080

The published GHCR image uses scriva serve --data /data, whose listeners default to loopback. Pass the address flags above when publishing Docker ports; binding 0.0.0.0 makes the server require an API key or TLS, and the example supplies an API key. The local image built by the root Dockerfile for Compose already bakes in the address flags and likewise requires an API key or TLS. See Getting Started for release archives, .deb/.rpm packages, Scoop and AUR.

---

Quick start

# 1. Start the server
scriva serve --data ./data --api-key dev-key

# 2. Insert a record
scriva-cli insert users '{"name":"alice","age":30}' --api-key dev-key

# 3. Query it
scriva-cli find users '{"field":"name","op":"eq","value":"alice"}' --api-key dev-key

Or use the interactive REPL:

scriva-cli repl --api-key dev-key

---

Run with Docker Compose

# Start the server (builds the image from the local Dockerfile on first run)
SCRIVA_API_KEY=dev-key docker compose up -d

# Insert a record against the REST gateway on :8080
curl -H "x-api-key: dev-key" \
  -d '{"data":{"name":"alice","age":30}}' \
  http://localhost:8080/v1/users/records

gRPC is exposed on :5433 and REST on :8080; data persists in the named scriva-data volume. See docker-compose.yml and the fully commented scriva.example.yaml for configuration.

---

What it is

ScrivaDB stores each collection as a set of NDJSON segment files — one JSON object per line, appended in order. There is no binary format, no external runtime, and no hidden state. You can inspect, backup, or migrate your data with any text editor or Unix tool.

Key properties:

• Exclusive directory lock — the engine takes an OS-level advisory lock (LOCK file) on the data directory to prevent corruption from concurrent writers; a second process trying to open the directory fails fast
• Fail-closed, verifiable recovery — open validates the persisted indexes against the segments (v2 coverage + fingerprints; replay only the unpersisted tail), refuses to start on damaged segments or ambiguous history (--integrity-policy fail), and scriva verify / scriva repair inspect and rebuild a stopped data directory with a verified backup first. See the recovery runbook
• Append-only writes — inserts, updates, and deletes are always new lines; no in-place modification
• Configurable durability — choose none (OS flush), always (fsync per write), or interval (fsync on a timer) to trade throughput against crash-loss window
• End-to-end integrity — every segment entry carries a CRC32C checksum, so silent on-disk bit-rot is caught on read instead of returning wrong data
• Encryption at rest (embedded) — transparent field- or record-level AEAD encryption (XChaCha20-Poly1305) configured through the embedded façade: scriva.WithPassphrase/WithEncryptionKey/WithKeyProvider + WithCollectionEncryption(EncryptFields/EncryptRecord). Values are opaque on disk under a reserved marker while reads return plaintext; a wrong key/passphrase fails fast on open; key rotation and enable/disable migrate existing data lazily or in bulk via a re-encrypting compaction pass. Segments, backups, and replication carry ciphertext for free. See docs/encryption-at-rest.md
• Background compaction — a goroutine per collection merges and deduplicates sealed segments; operators can also force a synchronous pass on demand (scriva-cli compact)
• Online backup — scriva-cli backup streams a consistent gzip snapshot of the live database; restore is a plain tar xzf into a data directory
• Leader→follower replication — start a hot standby with --replicate-from <leader>: the follower bootstraps from a snapshot then tails the leader's committed writes (each tagged with a monotonic global LSN), applying them through the normal write path so its indexes match exactly. Async (bounded lag), resumes after a disconnect from its persisted applied-LSN with no gaps or duplicates, and exposes (leader LSN, per-follower lag).

---

When to use it

ScrivaDB is the right tool when:

• You need persistence without standing up PostgreSQL or MongoDB
• Your data fits on one machine and you want human-readable files
• You're building CLI tools, local services, IoT daemons, or small web apps
• You want a simple HTTP/gRPC API you can call from any language

It is not the right tool for multi-node replication, complex joins, or datasets too large to compact on a single machine.

---

Embedding (use it as a Go library)

ScrivaDB's storage engine is a plain Go library, so you can skip the server entirely and run the database in-process — no gRPC, no network, no separate daemon. This is the right choice when your program is the only writer and you want ScrivaDB's durability and query model directly inside your binary.

go get github.com/srjn45/scriva/engine   # the storage engine
go get github.com/srjn45/scriva          # the ergonomic façade (recommended)

import "github.com/srjn45/scriva"

db, _ := scriva.Open("./data")            // embedded durability defaults (fsync ~1s)
defer db.Close()

sessions := db.MustCollection("sessions")
id, _, _ := sessions.InsertWithKey("sess-1", map[string]any{"status": "open"})
rec, _   := sessions.GetByKey("sess-1")   // caller-supplied string keys
_, _ = sessions.UpdateIfRev("sess-1", rec.Rev, map[string]any{"status": "closed"}) // CAS

Opt into transparent encryption at rest with a key option plus a per-collection policy — secrets are sealed on write and returned as plaintext on read, with nothing else in your code changing:

db, _ := scriva.Open("./data",
    scriva.WithPassphrase(os.Getenv("SCRIVA_PASSPHRASE")),
    scriva.WithCollectionEncryption("users", scriva.EncryptFields("password", "ssn")), // field-level
    scriva.WithCollectionEncryption("audit", scriva.EncryptRecord("id", "tenant")),     // record-level
)
users := db.MustCollection("users")
id, _, _ := users.Insert(map[string]any{"email": "[email protected]", "password": "hunter2"})
rec, _   := users.Get(id)          // rec.Data["password"] == "hunter2"; ciphertext on disk

A wrong passphrase fails fast on Open. Key rotation and enable/disable are runtime operations on the returned collection (RotateKey, SetEncryptionPolicy, MigrateNow, EncryptionStatus). See docs/encryption-at-rest.md for the full design.

The embedded surface includes caller-supplied string keys, per-record revisions with compare-and-swap, upsert, count/exists, secondary indexes, in-process Watch subscriptions, transparent encryption at rest, and a LoadJSONL bulk-import path. The engine pulls in no gRPC/protobuf/Prometheus/cobra/OpenTelemetry dependencies — a CI gate enforces that.

This is a distinct distribution channel: go get for embedding, while the standalone server ships via GHCR, tagged binary releases (with .deb/.rpm packages), and the Homebrew tap. Both build from the same repo. See docs/embedding.md for the full API reference, durability modes, the Watch overflow contract, the versioning/stability policy, and a migration guide.

---

Client SDKs

Idiomatic, hand-written client libraries are available for ten languages. Each wraps the same gRPC API, takes the same connection config (host, port, api_key, optional TLS CA cert), and wraps the core collection, record, index, single-collection transaction, Aggregate, Compact and Snapshot calls, including the streaming Find and Watch calls, plus the cross-collection transaction methods (BeginXTx, StageXTx, GetXTx, CommitXTx, RollbackXTx, XTxStatus). The replication/admin RPCs (Replicate, ReplicationStatus, Promote) have no hand-written SDK methods; call them through the generated gRPC stubs (each SDK bundles them) or, where a REST route exists, over REST.

Unknown commit outcomes. If CommitXTx fails with XTX_OUTCOME_UNKNOWN (gRPC UNKNOWN) or XTX_IN_PROGRESS (UNAVAILABLE), never retry the commit blindly. Call XTxStatus with the error's status_ref (or the idempotency key you chose at begin): COMMITTED means success, ABORTED/EXPIRED permits a new transaction, and PENDING means poll with backoff. Choose and keep an explicit idempotency key if a commit must be recoverable across a server restart. Inspect the scriva.xtx ErrorInfo reason, never the message text. Contract: docs/sdk-cross-collection-transaction-contract.md.

Verification status. The table below is what was actually run locally; "unverified" SDKs have the code and (where noted) tests but could not be executed by the release verifier, so treat their cross-collection transaction support as unverified until CI or a maintainer runs them.

SDK	XTx methods	Local verification (2026-10-20)
Python	yes	Unverified — pytest/grpcio not installed on the verifier; tests/test_xtx.py not run
JavaScript / TypeScript	yes	Verified — tests/xtx.test.ts passes, tsc --noEmit clean
PHP	yes	Unverified — no PHP runtime available; tests/test_xtx.php not run
Ruby	yes	Verified — test_xtx.rb passes (3 runs, 16 assertions)
Rust	yes	Unverified — build needs protoc (not installed); no XTx tests exist yet
C# / .NET	yes	Unverified — no .NET SDK available; only XTxErrorTests.cs exists, not run
Java	yes	Verified — XTxClientTest passes (ExampleTest needs a live server and was not run)
Kotlin	yes	Verified — ./gradlew test passes (8 tests)
Scala	yes	Verified — sbt test passes (7 tests)
Clojure	yes	Verified — lein test passes (7 tests, 30 assertions)

Versions. The SDKs are versioned independently of the server (currently 1.3.0, which adds the additive XTx methods; existing BeginTx/CommitTx/RollbackTx are unchanged). The XTx methods need a server that supports BeginXTx (v1.6.0+).

Language	Install	Reference
Python	pip install scriva	clients/python
JavaScript / TypeScript	npm i scriva	clients/js
PHP	composer require srjn45/scriva	clients/php
Java	io.github.srjn45:scriva-client (Maven Central)	clients/java
Kotlin	io.github.srjn45:scriva-client-kotlin:1.3.0 (Gradle, Maven Central)	clients/kotlin
Scala	"io.github.srjn45" %% "scriva-client-scala" % "1.3.0" (sbt, Maven Central)	clients/scala
Clojure	io.github.srjn45/scriva-client-clojure {:mvn/version "1.3.0"} (Clojars)	clients/clojure
Ruby	gem install scriva	clients/ruby
Rust	cargo add scriva	clients/rust
C# / .NET	dotnet add package Scriva.Client

Prefer to generate your own? The checked-in OpenAPI spec describes the 29 REST-annotated operations of the gateway. The five gRPC-only RPCs (BeginTx, CommitTx, RollbackTx, Snapshot, Replicate) are not in it; generate from proto/scriva.proto for those — see Getting Started.

---

Documentation

Document	Description
Getting Started	Install, run, first queries, TLS, config file, secondary indexes, metrics, logging, health probes
Index recovery runbook	Verify, repair and recover from index/data integrity problems
Architecture	Storage model, write/read paths, compaction, secondary indexes, crash safety
Embedding	Use ScrivaDB as an in-process Go library: scriva/engine API, keyed ops, CAS, Watch, migration, versioning policy

---

Build from source

git clone https://github.com/srjn45/scriva
cd scriva
make build          # produces bin/scriva and bin/scriva-cli
make test           # run tests with race detector
make proto          # regenerate gRPC code (requires buf)

---

Contributing

Contributions are welcome — see CONTRIBUTING.md for how to build, test, and submit changes.

---

License

MIT — see LICENSE.

---

Rebuilt from FileDB PHP — original college project, now in Go.