Second Brain OS
Kura browser captures now use the existing two-vault ingestion pipeline. See
capture and automation SOP for supported inputs, repeat
processing and source-change review.
An Obsidian second brain that reflects on its own corpus, cites everything it claims, and refuses to trust its own synthesis until an adversarial pass verifies it. Two vaults, and an append-only log of every crossing between them. Coding-agent-native — the compute is the session you already pay for.
Status: v0.3.0 — proactive reflection engine + hard VERIFY gate.
What this is
A bootable template for a working AI-augmented, two-vault Obsidian system. It ingests your AI chat exports, distills them into atomic notes, and — new in v0.3.0 — runs a scheduled reflection pass that asks questions of your own corpus and refuses to write an answer down until it's cited and adversarially checked. Retrieval is table stakes. The thing every other tool in this space skips is trustworthy synthesis you didn't ask for.
brain/ vault — MCP-wired. Your LLM (Claude Desktop, Claude Code, etc.) reads + writes here. Publishable.private/ vault — no MCP server points here, so nothing an LLM browses reaches it. The one exception is deliberate: /distill-inbox reads a raw conversation when you invoke it, through the pointer its stub names. Sensitive content lives here permanently.
Why this beats the field
Every row below is a structural gap, not a missing checkbox — see docs/reflection-engine.md for the full comparison and why each gap is architectural.
| This (SBO) | BASB (Tiago Forte) | Mem0 | Zep / Graphiti | Letta / MemGPT | Plain Obsidian + AI plugin |
|---|
| Proactive reflection | Yes — scheduled /reflect asks questions of the corpus unprompted | No — human runs Capture→Organize→Distill→Express by hand | No — extract-and-store only, no synthesis | No — recall is excellent, but on-demand | Partial — sleep-time agent compresses/merges, doesn't originate questions | No |
| Sovereign / local | Yes — local vault, local embeddings, nothing leaves the machine unless you export it | N/A (a methodology, not software) | No — managed cloud by default | Usually managed / hosted | Managed runtime | Yes, but no synthesis layer to be sovereign about |
| Cited-lens honesty | Every reflection carries {source_path, anchor} and is framed as a lens, not a verdict | N/A | No — facts stored, not framed | No — graph recall, not framed synthesis | No — merges without citation | No |
| Adversarial verify | Yes — distinct critic pass, 4 hard checks, failing candidates are dropped and logged | No | No | No | No | No |
| Privacy: two-vault split | Yes — filesystem boundary, MCP physically cannot resolve private/ | N/A | No — one store | No — one graph | No — one memory store | No — one vault, AI plugin sees everything |
Proven at scale
Built-in-public receipts, scale stats only:
- A 2-year Claude.ai export: 1,222 conversations / 11,447 messages ingested, 0 truncated.
- Every conversation's own
summary field (when the export carries one) is captured into brain frontmatter — new in 0.3, so agents read the AI's own compression instead of re-deriving it.
- A
memories.json export (the AI's pre-distilled cross-conversation memory) writes to a dedicated high-signal _memory-export.md that downstream agents are told to weight above any single conversation.
- 7 domain MOCs generated by a 12-agent, three-tier multi-model swarm (cheap audit → mid-tier build → deepest-tier adversarial verify) — verdict PASS, zero broken wikilinks after the fix pass. See
docs/swarm-curation.md.
- Verified through a 16-tool Obsidian MCP round-trip — every command in this repo (
/distill-inbox, /patterns-detect, /people-update, /sbo-verify, /reflect) runs against the same wired vault. See docs/obsidian-mcp.md.
- Test suite passing, zero-setup (
pytest works in a fresh clone with no pip install -e . — see What's new below).
30 minutes to wire.
Request your Claude.ai export first — it usually arrives in minutes, longer on a large account, and the download link expires 24 hours after it lands. Wire the system while you wait.
Quick start
git clone https://github.com/frankxai/second-brain-os
cd second-brain-os
# Install the ingestion package
python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e .
# Run setup
pwsh ./scripts/setup.ps1 # Windows
./scripts/setup.sh # macOS / Linux
See docs/getting-started.md for the full walkthrough.
The pipeline
Nine stages, each grounded in something you can point at on disk. No stage asserts anything that wasn't first retrieved from a real file. Full spec: docs/reflection-engine.md.
1 CAPTURE raw signal lands untouched — chat exports, daily notes, drops
v
2 INGEST dual-write applies the privacy boundary at the door
v private/ <- raw conversation, memories.json, audit row
v brain/ <- status:needs-summary stub + _INDEX.md corpus map
v
3 DISTILL /distill-inbox turns stubs into atomic notes (idea/learning/decision/…)
v
4 CONNECT /people-update refreshes _moc/ link-lists + per-person cross-refs
v
5 REFLECT *** new in v0.3.0 ***
v /reflect asks 3-5 cited questions of the whole corpus, weekly
v
6 SURFACE right memory, right moment — _moc/ + reflection output at session open
v
7 ACT highest-confidence verified insight -> content draft or next-action
v (human gates anything that goes external)
v
8 VERIFY *** hard gate, new in v0.3.0 ***
v citation-real | contradiction-queued | lens-framed | privacy-safe
v any FAIL drops the candidate — logged, never softened
v
9 COMPOUND reflection-log delta tells next week's run which errors to avoid
The three ingestion modes
sbo-ingest has three modes. Default is agent — recommended.
| Mode | Cost | Compute | When to use |
|---|
agent (default) | $0 extra | Your coding-agent session | You have Claude Code / ChatGPT / Cursor / Codex / Gemini open. Best quality (agent cross-references your vault). |
api | ~$0.005/convo on Haiku 4.5 | Anthropic API | Batch automation. No coding-agent session handy. |
dry-run | $0 | None — stubs only | Smoke-test the install. Verify the dual-write boundary. |
Default (agent mode) — recommended
sbo-ingest path/to/conversations.jsonl \
--brain-root /path/to/brain \
--private-root /path/to/private
# Writes raw to private/, stubs (status: needs-summary) to brain/_inbox/, refreshes _INDEX.md
ChatGPT exports can be passed as the downloaded ZIP without extracting it:
sbo-ingest path/to/chatgpt-export.zip \
--brain-root /path/to/brain \
--private-root /path/to/private
Both legacy conversations.json exports and current numbered
conversations-000.json, conversations-001.json, … shards are supported. ZIP
members are processed one shard at a time to keep peak memory bounded. Ambiguous,
nested, duplicated, mixed legacy/sharded, and gapped shard layouts fail closed.
Re-ingestion refreshes private raw sources atomically while preserving existing
same-conversation brain notes even after they move out of _inbox/. Completed runs write private receipts under
private/_distill/imports/.
If a memories.json sits next to the export (or you pass --memories path/to/memories.json), it's parsed and written to brain/_inbox/{platform}/_memory-export.md automatically — no separate command needed.
Then in Claude Code (or any other coding agent — see docs/cross-ai-portability.md):
/distill-inbox
The agent walks every stub, reads the linked raw conversation in private/, produces a real summary, writes it back, updates status: triage, and logs to private/_distill/audit.jsonl. No extra API spend.
API mode (optional)
sbo-ingest path/to/conversations.jsonl \
--brain-root /path/to/brain \
--private-root /path/to/private \
--mode api # or just set $ANTHROPIC_API_KEY
Smoke-test the install
sbo-ingest tests/fixtures/claude-ai-export-sample.jsonl \
--brain-root /path/to/brain \
--private-root /path/to/private \
--mode dry-run # legacy --dry-run flag also works
What's new in v0.3.0
/reflect — the reflection engine (Stage 5 of the pipeline). Weekly: gather the highest-signal 100 notes cheaply (frontmatter only, no per-note LLM call), generate 3-5 cited questions that cross-cut the corpus, synthesize each as a lens (never a verdict), run the VERIFY gate, write survivors to patterns/{YYYY}-W{ww}-reflection.md, log a reflection-delta so next week diversifies instead of looping. Spec: docs/reflection-engine.md.- VERIFY hard gate —
/sbo-verify gained a reflection mode: four independent checks (citation-real, contradiction-queued-for-human, lens-framing, privacy-safe) run as a distinct critic pass, never the same generation that wrote the candidate. Any single FAIL drops the candidate — it's logged, not softened. Every surviving artifact carries a sip_attestation YAML block recording what verified it and when.
memories.json + per-conversation summary ingestion — the export's own pre-distilled cross-conversation memory now lands as a dedicated high-signal _memory-export.md, and each conversation's own summary field (when present) is captured straight into brain frontmatter instead of being re-derived.- Cross-platform timestamp tolerance — malformed or out-of-range export timestamps are omitted as metadata instead of aborting an otherwise valid conversation import.
_INDEX.md corpus map — every ingest run regenerates a newest-first {date, title, summary} table per platform in brain/_inbox/{platform}/_INDEX.md, so an agent (or you) can read the whole corpus shape in one file instead of opening every stub.- Zero-setup tests — a root-level
conftest.py puts src/ on sys.path, so pytest collects and runs in a fresh clone without pip install -e . first (you still need the runtime deps installed — pip install -e . remains the one-command way to get those).
docs/obsidian-mcp.md — full MCP wiring guide: the HTTP-vs-HTTPS gotcha that silently breaks every tool call, the insecure-HTTP-off-by-default toggle, working config for Claude Code/Desktop, and a troubleshooting table.
What you get
~/second-brain/
├── brain/ # 10 community plugins, MCP-wired, agent-maintained zones
│ ├── _capture.md
│ ├── _inbox/{claude-ai,chatgpt,manual}/ # status: needs-summary lives here
│ │ └── _INDEX.md # newest-first corpus map, regenerated on every ingest
│ ├── notes/{ideas,learnings,decisions}/
│ ├── projects/
│ ├── people/ # auto-maintained by people-map agent
│ ├── patterns/ # weekly pattern-detector output + weekly *-reflection.md
│ ├── _meta/ # paid-tier psychometrics, businesses, decisions-history
│ ├── _moc/ # Maps of Content
│ └── _agents/ # agent prompt contracts
└── private/ # no MCP points here; read only via /distill-inbox
├── chat-history/{claude-ai,chatgpt}/ # raw conversations, UUID-named
├── journal/
├── relationships/
├── health/
├── finances/
└── _distill/
├── audit.jsonl # append-only ingest + distill log
├── imports/ # completed import receipts (members + counts)
└── pending/ # private patterns awaiting promotion to brain
Docs
Privacy
MCP is never pointed at private/. That is a path convention, not a
kernel-enforced boundary — there is no separate uid, container, or mount
namespace here, and a coding agent with shell access can read either tree.
So be exact about what this defends against: MCP scope creep, a misconfigured plugin, a retrieval tool pointed at the wrong root, and publishing brain/ by accident. It does not defend against the distilling agent itself, which you consciously grant one read per conversation. The audit log exists because the convention can fail.
Coding agents that run /distill-inbox read private/ once per conversation (with your explicit consent the moment you invoke the command), produce the summary, and never copy raw content into brain/. The audit log records every read. /reflect never reads private/ at all — if a candidate reflection would need private content to verify, it's dropped and the reason is stated, not silently skipped.
Titles cross the boundary by default
Filenames are hashed tokens, but the conversation title still lands as a stub's heading and as a row in _inbox/{platform}/_INDEX.md — one file listing every conversation you own. On a real 5,311-conversation export those titles named medical, legal, relationship and employer topics, in the vault an LLM reads. A title is often the most revealing line in a conversation.
Titles are also what make an inbox navigable, so this is your call:
SBO_TITLE_POLICY=redacted sbo-ingest path/to/export.zip
Headings and index rows fall back to the conversation token. The real title stays in private/, where the distilling agent still reads it.
Read docs/privacy-model.md before ingesting sensitive content. Run scripts/verify-privacy.{ps1,sh} weekly.
Composition
SBO is a vertical that composes SIP. It declines canon. The personal-instance pattern symlinks live commands and skills from your other substrates — see docs/composition-guide.md. The OSS template stands alone with no external dependencies beyond Python + Obsidian.
(Anthropic API is optional — needed only for --mode api.)
Testing
pytest -v
Test suite, zero-setup (root conftest.py puts src/ on sys.path): Claude.ai handler incl. per-conversation summary capture, ChatGPT JSON + fail-closed sharded ZIP handling, memories.json handling, identity-addressed idempotent dual-write, promoted-note preservation, path-containment guards, _INDEX.md corpus maps, private import receipts, all three ingest modes, audit logging, and zero-setup guards. CI runs the full matrix on ubuntu / macos / windows × Python 3.11 / 3.12 / 3.13.
License
MIT. See LICENSE.
Built on SIP
Starlight Intelligence Protocol v1.1.1. See Starlight-Intelligence-System.
Built by Frank Riemer. For builders, not consumers.
