leviathan

Leviathan is a single binary that turns records (JSONL, JSON, CSV/TSV, SQLite, or any database CLI's output) into a ranked full-text index. Agents ask in plain words and get short, cited result cards: ~450 tokens per answer at any dataset size, instead of grepping and reading raw history.

At 1M records (678 MB)	Leviathan	best grep strategy
Median tokens per question	436	107,122 (245×)
Relevant record returned	99.0% top 5 · 98.5% rank 1	96.0% within a 30K-char output
Worst case (1,200 questions)	602 tokens	9.7M tokens
Median latency	33 ms	92 ms

Measured on a synthetic maintenance log (one example dataset; nothing in Leviathan is specific to it). Methodology, all six scales and caveats: docs/BENCHMARKS.md.

Quickstart

cargo install leviathan-index       # or: cargo install --git https://github.com/elstongun/leviathan
cd examples/tickets && leviathan index
leviathan search -g acme "sso login loop after password reset"

leviathan search · customer C-ACME "Acme Corp" (7 tickets) · query "sso login loop after password reset" · shown 3 of 3 · 24 tickets indexed
[1] T-1001 · 2024-01-08 09:12 · rel 16.9
  Login loops back to sign-in page after password reset
  status: closed · priority: high
  resolution: Cleared stale session cookies on password reset; shipped in 4.2.1. Workaround: clear site data.
  match: Users who reset their password get redirected to the sign-in page again in an endless loop.
...

• -g accepts a key, name or partial name. Ambiguous or unknown groups list candidates and exit 3, never guess.
• No match in the group falls back to other groups, labeled OTHER CUSTOMER.
• Combine --where field=value, --since/--until, "phrases" and -exclusions.

Prebuilt binaries are on Releases. No runtime dependencies; the index is one SQLite file.

Map your data

Fields are paths (a.b, items[].name); only id is required.

Field	Enables
id	get, upsert, delete, citations
title / text	Card headline (weighted 2×) / searched text (default: all strings)
group / group_name	-g scoped search, name resolution, labeled fallback
date	--since, --until, recent
filters / display	--where facets / fields shown on cards
empty_values / rank.boost	Placeholders treated as missing / favor complete records

leviathan index ./export                      # everything in a folder: files of every format, every SQLite table
leviathan init ./export                       # infer a commented leviathan.toml
leviathan index tickets.csv --id "Ticket ID" --group customer_id --date created_at   # or flags
leviathan index -c tickets.toml               # or a config your agent wrote from the schema
leviathan describe                            # fields, groups, filter values, example calls

Any database works through its own CLI; Leviathan never holds credentials:

psql "$DATABASE_URL" -At -c "SELECT row_to_json(t) FROM tickets t" | leviathan index - -c tickets.toml
leviathan index app.db --sql "SELECT * FROM tickets" -c tickets.toml
duckdb -json -c "SELECT * FROM 'events/*.parquet'" | leviathan index - -c events.toml

A folder of tickets, customers and orders becomes one index with a kind per dataset: --where _kind=orders keeps one, ids read orders:1042, and a customer's group holds their tickets and orders together (several kinds). Messy exports are fine: wrapped or pretty-printed JSON, ; or | CSVs, byte-order marks, Latin-1, blank headers.

Builds are atomic and skipped when nothing changed; upsert and delete keep an index fresh. Full reference: docs/CONFIG.md.

Plug it into your agent

CLI + skill (recommended): any agent with a shell can call it. Copy skills/leviathan into ~/.claude/skills/ or paste it into AGENTS.md / .cursor/rules. Costs 0 tokens until used. skills/leviathan-memory does the same for memory.

MCP (optional): leviathan mcp serves five read-only stdio tools (search, resolve_group, get, describe, aggregate) whose descriptions include a summary of your dataset (~640 tokens per session). leviathan wrap <agent> prints the config for 24 agents (Claude Code, Codex, Cursor, VS Code / Copilot, Gemini CLI, Windsurf, Cline, Zed, JetBrains and more) and --apply writes it; leviathan wrap lists them all (docs/AGENTS.md).

Remote (optional): leviathan serve --http serves the same tools as Streamable HTTP MCP, REST and OpenAPI, with a bearer token or OAuth 2.1, so claude.ai, ChatGPT, the Anthropic and OpenAI APIs and agent frameworks can connect (docs/REMOTE.md).

Memory

Optional read/write memory for agents, in its own SQLite file. The agent remembers one claim at a time; anything that can change has a key, so the next write replaces the old value instead of contradicting it. Sessions start with a short briefing, and recall returns only what fits a token budget.

leviathan wrap claude --memory --hooks --rules --apply   # tools, session-start briefing, always-on rule
leviathan memory remember "Prefers pnpm over npm" --kind preference -s sam -k package_manager
leviathan memory recall package manager
leviathan memory import ~/.claude/projects/<project>/memory/MEMORY.md   # bring an existing markdown memory

After 300 sessions	Leviathan memory	MEMORY.md, whole file	MEMORY.md, first 200 lines
Tokens at session start	736	33,313	2,588
Answer in context	100%	100%	18%
Stale value in context	0%	73.8%	36.4%

Writes are explicit (no model runs inside Leviathan), secrets are refused, and history stays queryable (as_of, history). Guide, formats and benchmark: docs/MEMORY.md.

Commands

Command	Does
init / index / upsert / delete	Propose a mapping / build / update / remove records
search [-g G] [words]	Ranked records (--scope, --where, --since, --until, -n, --offset)
aggregate [--by B] [--metric M --field F]	Exact counts or sum/avg/min/max over every matching record, in total or per group, year/month/day or filter value (-g, --group-contains, --where, --since, --until, -n)
recent [-g G] · resolve <g> · get <id…> · describe	Newest · group candidates · full records · index summary
memory remember / recall / forget / briefing	Write / ranked read / retire / session-start briefing (--memory=PATH)
memory list / history / stats / import / export / prune	Browse / slot versions / counts / markdown or JSONL in / backup / clean up
mcp [--memory] [--remote URL] · serve --http ADDR	stdio MCP server or bridge · HTTP MCP + REST server
wrap [agent]	Agent config (--memory, --hooks, --rules, --apply); no agent lists targets

Global: --index PATH, --json, --max-chars N. Exit codes: 0 ok (zero hits included), 1 error, 2 bad request, 3 group unknown/ambiguous.

How it works

Records stream into one SQLite file with an FTS5 index. A search resolves the group (exact → name → contains → fuzzy), runs one FTS5 match where group and filter values are indexed tokens (no post-filtering), ranks by BM25 × boosts, and decodes only the top N into capped cards. A record's own group name is excluded from scoped matching, placeholders count as missing, and every answer reports shown N of M so agents can tell "no match" from "no data".

Reproduce the benchmark

python3 -m venv bench/.venv && bench/.venv/bin/pip install -r bench/requirements.txt
bench/.venv/bin/python bench/run_bench.py && bench/.venv/bin/python bench/report.py   # ~30 min, ~4 GB

Contributing, security, license

See CONTRIBUTING.md (ranking changes need before/after benchmark numbers) and SECURITY.md (indexes read-only, offline unless you serve or connect over HTTP, unsafe-free; report privately). Licensed under Apache-2.0; dependencies in THIRD_PARTY.md.