#+TITLE: Crucible: LLM-Compiled Knowledge Base #+AUTHOR: John Kitchin #+DATE: [2026-04-03 Fri]

[[./crucible.png]]

• Overview

Crucible is a personal knowledge base system where primary sources (papers, notebooks, data) are ingested and distilled by an LLM into a structured, interlinked org-mode wiki. The wiki is the LLM's domain: you rarely edit it directly. Instead, you converse with Claude, and it reads, writes, and maintains the wiki on your behalf.

The core idea, inspired by Andrej Karpathy's LLM knowledge base pattern, is that raw data from various sources is collected, then compiled by an LLM into a wiki of org-mode files, then operated on by various CLI commands to do Q&A and to incrementally enhance the wiki. Everything is viewable in Emacs via scimax.

Key design principles:

• Separation of concerns: primary sources (which may be copyrighted) are kept separate from distilled content (which is shareable).
• Org-mode native: all wiki articles use org-mode with scimax conventions, org-ref citations, properties drawers, and filetags.
• Graph database: a SQLite database tracks the relationships between sources, articles, concepts, and links, enabling structured queries that grep cannot do.
• Hybrid search: combines FTS5 keyword search with ollama vector embeddings for meaning-based retrieval, merged via reciprocal rank fusion.
• Interoperability: multiple crucible instances can be peered for cross-querying via SQLite's separate connection mechanism.
• Knowledge evolution: derivation chains track how articles are synthesized from other articles over time.

• Installation

From PyPI:

#+BEGIN_SRC bash pip install crucible-llm #+END_SRC

Or from source for development:

#+BEGIN_SRC bash git clone https://github.com/jkitchin/crucible.git cd crucible pip install -e . #+END_SRC

Then initialize a new crucible project and wire it into Claude Code:

#+BEGIN_SRC bash cd /path/to/your/project crucible init --root . crucible install # installs Claude Code skill + CLAUDE.md directive #+END_SRC

The =install= command does three things:

1. Copies the skill into =~/.claude/skills/crucible/=
2. Generates a manifest summarizing the knowledge base contents
3. Adds a directive to =~/.claude/CLAUDE.md= so agents know to check the crucible before answering domain-specific questions

=init= registers the project and installs the skill, but only =install= writes the =~/.claude/CLAUDE.md= directive. There is one directive, so a new crucible does not take it over until you run =install= in it.

To remove: =crucible uninstall=.

** Installing the skill with knowhow

The skill lives in =skills/crucible/SKILL.md= and follows the [[https://agentskills.io/specification][Agent Skills specification]], so it can also be installed with [[https://github.com/jkitchin/knowhow][knowhow]] (or =npx skills=) for Claude Code and other agents (VS Code, Codex, OpenCode, Gemini CLI, Cursor):

#+BEGIN_SRC bash uvx knowhow add jkitchin/crucible --skill crucible # this project uvx knowhow add jkitchin/crucible --skill crucible -g # global

or: npx skills add jkitchin/crucible --skill crucible

#+END_SRC

The skill still needs the =crucible= CLI on your PATH. Installing it this way does not generate the manifest or the =~/.claude/CLAUDE.md= directive, so run =crucible install= inside your project if you want those too. Contributors can check the skill with =uvx knowhow validate .=.

For semantic search, you also need ollama running with an embedding model:

#+BEGIN_SRC bash ollama pull nomic-embed-text crucible embed #+END_SRC

• Directory Layout

A crucible project keeps everything in a =.crucible/= directory at its root:

#+BEGIN_EXAMPLE your-project/ .crucible/ CLAUDE.md Conventions for agents working in the wiki .gitignore Excludes crucible.db and sources/external/ crucible.db Wiki graph database (SQLite, regenerable) references.bib Bibliography of ingested sources sources/ external/ Copyrighted material (gitignored) pdfs/ Academic papers, reports web/ Web clippings notebooks/ Your own ELN entries (shareable) data/ Your own experimental data (shareable) wiki/ index.org Auto-generated master index MANIFEST.md Summary for agent discovery concepts/ One topic per file summaries/ One per ingested source comparisons/ Cross-source analysis methods/ Technique and methodology articles #+END_EXAMPLE

The global registry of all crucibles on the machine is =/.crucible/registry.json=. Because =/.crucible/= is the registry, the home directory itself cannot be a crucible project.

This repository holds the Python package (=src/crucible/=), the agent skill (=skills/crucible/SKILL.md=, Agent Skills spec), the documentation (=docs/=) and the tests (=tests/=).

• Quick Start

See [[file:docs/tutorial.org][the tutorial]] for a complete walkthrough. The short version:

#+BEGIN_SRC bash

Ingest a source

crucible ingest paper.pdf

Or from the web

curl -sL "URL" | pandoc -f html -t org | crucible ingest - -t "Title" --type web

Or a batch

crucible ingest "papers/*.pdf"

Ask Claude to distill it

(Claude reads the source, proposes articles, you approve, it writes them)

Sync the database

crucible sync

Generate embeddings for semantic search

crucible embed

Search

crucible search "activation energy" # hybrid (auto) crucible search "how does temperature affect rate" # finds by meaning crucible search "catalyst" --mode fts # keyword only

Navigate

crucible concepts # list all topics crucible concept thermodynamics # articles on a topic crucible backlinks wiki/concepts/activation-energy.org crucible related wiki/concepts/activation-energy.org crucible history wiki/comparisons/synthesis.org # derivation chain

This is the beginning of the README. Read the rest at the original. Read the full README ↗