Chicory

A self-organizing memory and associative network (MAN) for LLMs. Stores memories with semantic embeddings and tags, then discovers co-occurrence, semantic, asymmetric semiotic, and glyph-structural relationships through a Prime Ramsey Lattice projected onto a Poincaré disk. Each chicory MAN maintains a four-network tag relational tensor where each channel activates from a distinct evidence source — ingest-time co-occurrence, query-time co-retrieval, or structural resonance — with data-derived lateral inhibition (confident anti-association) and glyph-aware cross-referencing. Bootstraps from zero with no seed data. An episodic memory-to-memory tensor, forest block organizer, and canopy growth layer emerge from use. A directional canopy tracks retrieval causality — inflow canopy identifies convergence attractors in memory-space, outflow canopy identifies distributor neighborhoods in tag-space — and feeds back into bridge hub penalties. Temporal tag episodes track drift in tag-space and detect when conversation returns to a previous topic region — episodes form a revisitable graph, not a timeline. Association strengths are reweighted on every retrieval through centroid sub-graph dynamics — no passive time-decay, no arbitrary caps. Retrieval runs two independent paths — FAISS embedding search and tag-graph navigation (TagSpace) — with an in-memory tensor cache eliminating all tensor I/O. Warm queries complete in ~35ms.

Install

pip install chicory-man[mcp]

With additional providers and features:

# Anthropic + OpenAI LLM support
pip install chicory-man[all-llm,mcp]

# Web dashboard
pip install chicory-man[dashboard]

# Glyph system (ByT5 encoder for character-level lattice embeddings)
pip install chicory-man[glyph]

# REST API server
pip install chicory-man[api]

# Cross-project signal federation
pip install chicory-man[commons]

# Everything
pip install chicory-man[all-llm,mcp,dashboard,glyph,api,commons]

Or without any extras:

pip install chicory-man

OpenAI / ChatGPT Setup

Chicory can use OpenAI's current flagship ChatGPT/API model through the OpenAI Responses API. Configure the OpenAI provider and model with environment variables:

$env:OPENAI_API_KEY = "sk-..."
$env:CHICORY_LLM_PROVIDER = "openai"
$env:CHICORY_LLM_MODEL = "gpt-5.5"
chicory chat

For a persistent .env setup:

OPENAI_API_KEY=sk-...
CHICORY_LLM_PROVIDER=openai
CHICORY_LLM_MODEL=gpt-5.5

Grok / Gemini Setup

Grok and Gemini use the same openai Python package through each provider's OpenAI-compatible API. Pick one provider at a time with CHICORY_LLM_PROVIDER.

# xAI Grok
XAI_API_KEY=xai-...
CHICORY_LLM_PROVIDER=grok
CHICORY_GROK_MODEL=grok-4.20-reasoning

# Google Gemini
GEMINI_API_KEY=...
CHICORY_LLM_PROVIDER=gemini
CHICORY_GEMINI_MODEL=gemini-2.5-flash

MCP Server Setup (Claude Code / Claude Desktop)

1. Set your API key and start the server:

export ANTHROPIC_API_KEY=sk-ant-...
chicory-mcp

2. Add to your Claude Code config (~/.claude.json):

{
  "mcpServers": {
    "chicory": {
      "type": "stdio",
      "command": "chicory-mcp",
      "env": {
        "ANTHROPIC_API_KEY": "sk-ant-..."
      }
    }
  }
}

The embedding model (all-MiniLM-L6-v2) downloads automatically on first use. The database is created at ~/.chicory/chicory.db by default (override with CHICORY_DB_PATH).

MCP Tools

The server exposes 12 tools:

Tool	Purpose
store_memory	Store a memory with tags and importance
retrieve_memories	Semantic/hybrid/tag search with tensor-boosted recall
deep_retrieve	Recursive retrieval following association chains into the deep past
get_trends	Tag trend signals: level, velocity, jerk, temperature
get_phase_space	Tag positions on the trend-temperature × retrieval-frequency plane
get_synchronicities	Detected synchronicity events with effective strength and reinforcement
get_meta_patterns	Higher-order recurring themes across synchronicity events
get_lattice_resonances	Prime Ramsey lattice: angular positions, resonances, void profile
ingest_codebase	Scan source files and store structural summaries (AST-based)
ingest_documents	Ingest full text content of files, chunked for retrieval
process_signals	Convert pending cross-project signals into commons memories
get_commons_log	Chronological history of federation activity

REST API & Docker

Chicory includes a FastAPI web server for browser-based demos with per-session isolated databases:

# Local
chicory-api

# Docker
docker compose up -d

The API server runs on port 8000 with password-based auth (set CHICORY_PASSWORD in .env).

Quick Start (CLI)

# Interactive chat with your configured LLM + memory
chicory chat

# System status
chicory status

# Ingest a file or directory
chicory ingest path/to/docs/ --recursive

# Watch a directory for changes
chicory watch path/to/docs/

# Re-embed all memories with a new model
chicory reembed --model all-mpnet-base-v2

# Run a full model migration
chicory migrate

# Backfill single-letter composition tags
chicory backfill-letters

# Launch the web dashboard
chicory dashboard

# Commons federation commands
chicory commons --help

Demo

# Run the architecture demo (no API keys, no model downloads, just numpy)
pip install chicory-man[dev]
python -m examples.ramsey_network_demo

The demo walks through 9 phases from blank slate to self-organizing memory in ~0.2 seconds, entirely in-memory with mock embeddings. Pass any file to demo document ingestion:

python -m examples.ramsey_network_demo path/to/any/document.txt

Supports .txt, .md, .py, .json, .csv, .pdf, .docx, and 30+ other formats.

Architecture

                        store_memory()              retrieve_memories()
                             |                             |
                             v                             v
                  +---------------------+       +---------------------+
  Layer 1         |    Memory Store     |       |  Semantic / Hybrid  |
                  |  tags, embeddings,  |       |     Retrieval       |
                  |  salience scoring   |       |  (FAISS + tensor)   |
                  +---------------------+       +---------------------+
                             |                        |          |
                    tag assignment             FAISS path   TagSpace path
                             v                   |          |
                  +---------------------+    +---------------------+
  Layer 2         |   Trend Engine      |    | Retrieval Tracker   |
                  | level, velocity,    |    | frequency, tag hits |
                  | jerk, temperature   |    |                     |
                  +---------------------+    +---------------------+
                             \                           /
                              \                         /
                               v                       v
                         +---------------------------+
  Layer 3                |      Phase Space          |
                         | (temperature, retrieval)  |
                         |  -> quadrant per tag      |
                         +---------------------------+
                                      |
                            statistical anomalies
                                      v
                         +---------------------------+
  Layer 3                | Synchronicity Detector    |
                         |  - dormant reactivation   |
                         |  - cross-domain bridges   |
                         |  - semantic convergence   |
                         +---------------------------+
                                      |
                           SynchronicityEvent objects
                                      v
                  +-------------------------------------------+
  Layer 3         |              TagSpace                     |
                  |  independent tag-graph retrieval path     |
                  |  lexical match → tensor fan-out →         |
                  |  inward ratio → memory resolution         |
                  |  hub filtering for context tag pruning    |
                  +-------------------------------------------+
                                      |
                                      v
                  +-------------------------------------------+
  Layer 3.5       |          Prime Ramsey Lattice             |
                  |  angle = PCA(tag_centroids) -> atan2      |
                  |  slot(p) = floor(angle * p / 2pi) % p    |
                  |  resonance = shared slots across primes   |
                  +-------------------------------------------+
                                      |
                              resonance pairs
                                      v
                  +-------------------------------------------+
  Layer 3.5       |      Tag Relational Tensor               |
                  |  4 independent networks per tag pair:     |
                  |    co-occurrence  (PMI, ingest+queries)   |
                  |    semantic       (Poincaré, queries only)|
                  |    semiotic       (P(B|A), ingest+queries)|
                  |    glyph          (resonance, structural) |
                  |  + lateral inhibition (anti-association)  |
                  |  + in-memory TensorCache (~6MB, all I/O)  |
                  +-------------------------------------------+
                                      |
                           each retrieval triggers
                                      v
                  +-------------------------------------------+
  Layer 3.5       |      Centroid Sub-Graph Reweighting       |
                  |  tag centroids (EMA on store)             |
                  |  co-retrieval edges (EMA on retrieve)     |
                  |  add incoming + subtract inverted         |
                  |  parallelness-gated (orthogonal preserved)|
                  +-------------------------------------------+
                        |                           |
                        v                           v
           +------------------------+   +------------------------+
           |   Poincaré Disk        |   |   Glyph Ramsey Lattice |
           |  hyperbolic projection |   |  ByT5 character-level  |
           |  of lattice positions  |   |  structural resonance  |
           +------------------------+   +------------------------+
                                      |
                                      v
                  +-------------------------------------------+
  Layer 4         |          Episodic Tensor                   |
                  |  sparse memory-to-memory edge cache        |
                  |  gateways: semantic threshold, co-retrieval,|
                  |    same block, tag affinity threshold       |
                  |  lifecycle driven by composite strength     |
                  |  no arbitrary caps (super-hubs allowed)     |
                  +-------------------------------------------+
                                      |
                                      v
                  +-------------------------------------------+
  Layer 4         |       Temporal Tag Episodes                |
                  |  drift-detected tag-space clusters          |
                  |  adaptive threshold (Welford variance)      |
                  |  revisitable: returns to old topic region   |
                  |  sync events can force episode boundaries   |
                  |  episodes form a graph, not a timeline      |
                  +-------------------------------------------+
                                      |
                                      v
                  +-------------------------------------------+
  Layer 4         |    Forest (co-occurrence + bridge base)    |
                  |  co-occurrence optimizer: compress local    |
                  |  bridge optimizer: preserve traversability  |
                  +-------------------------------------------+
                                      |
                                      v
                  +-------------------------------------------+
  Layer 4.5       |    Canopy (emergent growth layer)          |
                  |  discovers memory clusters from episodic   |
                  |  co-activation + bridge edges               |
                  |  never decays — only grows                  |
                  +-------------------------------------------+
                                      |
                                      v
           +------------------------+   +------------------------+
  Layer 4.5|   Inflow Canopy       |   |   Outflow Canopy       |
  (direct.)|  convergence attractors|   |  distributor tag hoods |
           |  in memory-space       |   |  in tag-space          |
           |  diversity of incoming |   |  diversity + reach of  |
           |  query contexts        |   |  result clusters       |
           +------------------------+   +------------------------+
                        \                       /
                         \   directional_block_scores
                          \       |       /
                           v      v      v
                  +-------------------------------------------+
  Layer 4.5       |  Bridge Hub Penalty (directional rescue)  |
                  |  degree penalty * inflow rescue            |
                  |                 * outflow rescue           |
                  +-------------------------------------------+
                                      |
                                      v
                  +-------------------------------------------+
  Layer 4         |  Meta-Analysis & Adaptive Thresholds      |
                  |  threshold EMA, burn-in, pattern clusters  |
                  +-------------------------------------------+

The Four Tensor Networks

Each tag pair (A, B) carries four independent strength signals, each with a distinct activation story:

Co-occurrence (ingest + queries) — Pointwise Mutual Information computed from both memory co-occurrence and retrieval co-occurrence:

PMI_mem(A,B)  = log( P_mem(A,B)  / (P_mem(A)  * P_mem(B))  )   # ingest-time
PMI_ret(A,B)  = log( P_ret(A,B)  / (P_ret(A)  * P_ret(B))  )   # query-time
strength      = PMI_mem + PMI_ret

Memory PMI measures how surprising a co-occurrence is in the corpus. Retrieval PMI reinforces pairs that are actually used together in practice — tags that appear in the same query results accumulate evidence independent of how they were stored.

Semantic (queries only) — Poincaré geodesic distance between tag embedding centroids, gated by co-retrieval evidence. This channel starts dark and only activates for tag pairs that have been retrieved together at least once:

strength = exp(-hyperbolic_distance(centroid_A, centroid_B))
gate     = co_retrieval_count > 0

The co-retrieval gate means manual sync cannot create semantic connections that haven't been validated by actual use. This is the tensor's learning signal — it lights up as the system is used.

Semiotic (ingest + queries) — Directed associative strength via conditional probability from both memory and retrieval co-occurrence:

semiotic_forward  = P_mem(B|A) + P_ret(B|A)
semiotic_reverse  = P_mem(A|B) + P_ret(A|B)

These are naturally asymmetric. If "topology" appears in 1 retrieval and "mathematics" in 5, with 1 shared: P(mathematics|topology) = 1.0, but P(topology|mathematics) = 0.2. Memory co-occurrence provides the base directionality; retrieval patterns reinforce or shift the asymmetry through use.

Glyph (structural) — Lattice resonance strength from the glyph Ramsey lattice. Two tags resonate when they share slot positions across multiple prime scales despite being semantically distinct. Strength = sum(log(p)) for shared primes, measuring information-theoretic surprise. This channel requires no usage evidence — it captures structural relationships invisible to embeddings.

All four networks are stored in a single tag_relational_tensor table with a CHECK(tag_a_id < tag_b_id) constraint. The semiotic layer encodes directionality as semiotic_forward (a→b) and semiotic_reverse (b→a) within this symmetric key.

Channel Activation Summary

Channel	Ingest (memory)	Queries (retrieval)	Structural
Co-occurrence	PMI	PMI	—
Semantic	—	co-retrieval + Poincaré	—
Semiotic	P(B|A)	P(B|A)	—
Glyph	—	—	resonance
Inhibition	confident co-occurrence deficit (negative PMI)	—	—

Lateral Inhibition

Inhibition is the other half of the ingest co-occurrence evidence. The co-occurrence channel keeps the pairs that appear together more often than chance; inhibition is built from the pairs that are each common but appear together less often than chance — alternatives such as python and javascript, or postgres and mongodb. When the context is clearly about one of them, partners that belong to the other are turned down. This mirrors lateral inhibition in biological neural networks — active neurons suppress their competitors to sharpen contrast.

Evidence is counted per unit, COALESCE(source_path, id), so the chunks of one document count once. Only non-archived, non-quarantined memories whose tags were chosen by a tagger are evidence, and only the tags it chose: file-ingested memories (content_hash set) are left out, because their tags are their path (extension and directory tags are mutually exclusive by construction); the rows the store adds itself (assigned_by = 'system': the split words of a compound tag, which copy its incidence exactly, and the temporal and letter tags) are skipped; and temporal (month-, day-, …), single-letter and digit tags never count, whoever applied them. Over N units, with n_a units carrying tag a and O units carrying both:

E(a,b)  = n_a · n_b / N                        # expected co-occurring units
p(a,b)  = P(Poisson(E) ≤ O)                    # tested only where E > −ln(fdr) ≈ 3.0
select    Benjamini-Hochberg at fdr = 0.05     # R discoveries among m testable pairs
q*      = 1 − R · fdr / m                      # FCR-adjusted confidence
I(a,b)  = clip(1 − Γ⁻¹(O + 1, q*) / E, 0, 1)   # confident missing fraction

Γ⁻¹(O + 1, q*) is the exact one-sided Poisson upper limit on the observed count, so I is the fraction of the expected co-occurrences that are confidently missing. It is exactly 0 for rare tags (with E ≤ 3 no amount of absence is significant), for any pair seen as often as chance, and — because selection controls the false-discovery rate — for a store with no real anti-association. A new store starts with no inhibition and grows it only as evidence accumulates. Lexical variants (test/testing, postgres/postgresql) are exempt. Pairs live in their own sparse tag_inhibition table, independent of glyph resonance: tensor rows exist only for glyph-resonant pairs, which competing tags almost never are.

At retrieval, every active context tag inhibits every partner — no tensor edge between them is needed. Each tag carried by the FAISS top-5 results is weighted by its share of them (a_s, taken before hub filtering, which prunes excitation seeds, not inhibition evidence), and the inhibitors of a partner p are pooled as a noisy-OR, so H stays below 1 however large the context:

H_p = 1 − Π_s (1 − a_s · I(s,p))     # over active context tags s
R_p = E_p / (1 + β · H_p)            # β = tensor_inhibition_weight = 2.0

The suppression is divisive (shunting): it matters at any excitation scale, never flips a sign, never drops a partner, and never divides one by more than 1 + β. Where a co-retrieval gate decides which partners take part (resonant memory lookup, glyph association scoring), inhibition is applied after it, so membership is decided on excitation alone and inhibiting one partner can never open the gate to others. With no evidence R = E exactly, and β = 0 or inhibition_enabled = false (CHICORY_INHIBITION_ENABLED=false in the environment or .env) switches it off exactly; a negative or non-finite β reads as off rather than dividing by zero or flipping a sign. It applies to resonant memory lookup, glyph association scoring, and TagSpace lexical scoring (the lexical seeds inhibit the tags they fan out to); the episodic tensor and Ramsey square scoring divide by 1 + β·H the same way.

Recompute is never on the store path. A background phase recounts once memory_tags has grown 2% past the watermark saved in librarian_state (no more than once a minute), with the foreground lock released while it counts; the result is diff-written in small transactions that yield to a waiting request and resume on the next pass, then swapped into the TensorCache whole. chicory sync (run_sync) recomputes synchronously. Other processes on the same database (one MCP server per client) see the new watermark on their next check and reload their index, and a plan that a recompute or a tag merge overtook while it was counted or queued is dropped rather than written. A merge deletes the rows touching both tags and replaces the watermark with a reset one, so every process recounts on its next pass; a watermark counted under an older definition of evidence reads as absent, so an upgrade that changes what counts recounts too. Retrieval co-occurrence is deliberately not evidence in either direction: inhibition changes what gets co-retrieved, so it would feed on its own output and lock in, and FAISS co-retrieves substitutes precisely because they are similar.

There is no geometric (antiparallel) gate. Competitors are usually neighbors in embedding space, not opposites — antiparallel pairs are already kept apart by FAISS — and the old parallelness (cosine of two tags' glyph-lattice angle difference) was above 0.88 on every resonant row, so the gate never opened. The tensor columns parallelness and inhibition_strength are retired: kept in the schema for compatibility, never read or written. The centroid sub-graph's parallelness factor (below) is a different quantity and is unchanged.

Centroid Sub-Graph Reweighting

Synchronicity and resonance strengths are not passively decayed by time. Instead, every retrieval actively reweights existing scores based on what the system is actually doing:

On store: Tag centroids are updated via exponential moving average (EMA). Each tag's centroid is a unit-normalized running average of all memory embeddings assigned to it.

On retrieval: The system computes incoming association strengths for every pair of activated tags:

incoming(A,B) = cosine_sim(centroid_A, centroid_B) × mean_relevance × scale

These incoming strengths are then ranked highest to lowest and the values are inverted (reversed). Each pair receives:

• Addition: the raw incoming strength (boosting active associations)
• Subtraction: the inverted value × parallelness factor (suppressing redundant patterns)

The parallelness factor prevents collateral damage to independent associations. Each pair defines a direction vector in embedding space (normalize(centroid_A - centroid_B)). The subtraction is scaled by how geometrically aligned the pair is with any stronger pair:

parallelness(i) = max(|cos(direction_i, direction_j)|)  for all j ranking above i

• Parallel weak pairs (redundant with a stronger pattern): full subtraction — prevents rigidity
• Orthogonal weak pairs (independent patterns): near-zero subtraction — preserved
• Dominant pair (rank 0): no reference above it → zero subtraction → pure additive boost

The net effect: strongest incoming associations grow, weakest parallel ones shrink, and orthogonal associations pass through unharmed. Old connections that stop being actively retrieved lose strength as new retrievals erode them.

Dual Retrieval Paths

Retrieval runs two independent paths that merge into a single scored candidate set:

FAISS Path — Embeds the query, searches the FAISS index for the top-K nearest neighbors by cosine similarity. These candidates are then re-ranked through candidate-scoped tensor scoring: resonant memory lookup, glyph associations, and semiotic convergence — all operating on the in-memory tensor cache, scoring only the ~400 FAISS candidates.

TagSpace Path — Runs before embedding (instant). Tokenizes the query, matches tokens against an in-memory tag name index, expands seed tags through the tensor graph (fan-out), and scores results by inward ratio (convergence × inward fraction). Finds memories that share tag-graph structure with the query even when embedding similarity is low.

Both paths contribute weighted scores to the same scores dict. TagSpace-discovered memories that weren't in FAISS results get scored by tag-graph signal alone. FAISS candidates get both signals.

In-Memory Tensor Cache

The TensorCache loads the entire tag_relational_tensor (~74K rows, ~6MB) into Python dicts indexed by tag_a_id and tag_b_id. All tensor queries during retrieval become dict lookups (microseconds) instead of random page reads across a 10GB SQLite file (seconds). The cache is lazy-loaded on first access and invalidated after any tensor write. It also holds the lateral inhibitor index from tag_inhibition, which has its own lifecycle: warmed once at boot and swapped whole after each recompute (this process's, or another process's seen through the shared watermark), never reloaded by a store.

Hub Filtering

Context tags from FAISS top-5 results are filtered by seed edge count before downstream re-ranking. Tags whose degree to the combined lexical + centroid seed set exceeds tag_space_max_context_seed_edges are pruned as hubs — they connect to everything and add noise to resonant lookup and semiotic convergence.

Glyph System

The glyph system adds a parallel character-level lattice that captures structural relationships between tag names — relationships invisible to semantic embeddings.

Glyph Lexicon — A hardcoded mapping of ~80 concepts to Unicode symbols (e.g., Memory → ☍, Time → ⧖, Transformation → ⫚). Includes pair relationships (oppositions, transformations, co-occurrences) derived from GPT-GU's semiotic framework.

Glyph Encoder — A ByT5 (byte-level T5) encoder that produces d_model-dimensional embeddings of tag name strings at the character level. These are projected to 2D via PCA for angle computation on a separate glyph Ramsey lattice.

Glyph Bridge — Translates Chicory tag names to GPT-GU glyph symbols via the lexicon or ByT5 generation, then embeds the glyph symbols (not the English words) through ByT5. Lattice embeddings land in GPT-GU's symbolic space, where structural glyph relationships surface as Ramsey resonances.

Glyph Analyzer — Scans memory content for concept words and maps them to glyphs, detecting concept pairs (e.g., "X vs Y") for relational tagging. Glyph definitions are automatically injected into MCP tool responses.

When the glyph system is active, retrieval scores receive a configurable boost (glyph_retrieval_boost_weight) for memories whose tags share glyph associations.

Poincaré Disk Projection

Lattice positions are optionally projected into the Poincaré disk model of hyperbolic space (H²). Angular coordinates come from PCA projection of embedding centroids; radial depth encodes hierarchical specificity derived from prime slot population density.

• Events with slots in many primes (general themes) sit near the disk center
• Events with slots in few primes (specific/niche topics) sit near the boundary
• Geodesic distance (arccosh metric) gives a natural measure of relatedness that respects the hierarchy
• Einstein midpoint computes weighted hyperbolic centroids for cluster analysis

Five-Tier Architecture: Tag Tensor → Episodic Tensor → Temporal Episodes → Canopy → Directional Canopy

The system maintains five tiers of relational structure at different granularities:

Tag Tensor (global, abstract) — The four-network tensor described above. Operates on tag pairs — there are relatively few tags (~thousands), so the tensor can track all resonant pairs. Answers: "How do concepts relate globally?"

Episodic Tensor (local, concrete) — A sparse memory-to-memory edge cache. Memories are many (~tens of thousands), so edges are materialized lazily through gateway rules:

1. FAISS top-K: semantic neighbors in embedding space
2. Co-retrieval: memories that appeared in the same query results
3. Same block: memories in the same forest block
4. Tag affinity: high projected affinity from the tag tensor

Each edge carries a composite strength = semantic + tag_projected + co_retrieval + bridge, where tag_projected is the mean of all four tensor channels divided by 1 + β·H (H combines the anti-association between the two memories' tags as a noisy-OR). All thresholds are derived from the composite signal:

• Gateway: semantic neighbors above similarity_threshold (FAISS range search), plus co-retrieval, same-block, and tag affinity above episodic_tag_affinity_threshold
• candidate → warm: composite >= episodic_tag_affinity_threshold
• warm → mature: composite >= midpoint of the threshold and 1.0
• mature → protected: exempt from weak-edge pruning and decay

No arbitrary K or per-memory caps. Edge discovery, lifecycle promotion, and pruning are all driven by the same composite signal — the combined mathematical evidence from embeddings, tensor channels, co-retrieval history, and bridge structure. Heavily-connected memories form natural super-hubs. Edges that stop being activated decay and are eventually archived.

Temporal Tag Episodes (temporal, revisitable) — Tracks drift in the tag-space centroid over time and segments activity into episodes. Each episode is a contiguous period where stores and retrievals operate in a coherent region of tag space:

• Drift detection: On each store or retrieval, compute the mean of the operation's tag centroids (reusing centroid sub-graph embeddings). Compare cosine distance to the current episode's centroid. If distance exceeds an adaptive threshold → snapshot the current episode and start a new one.
• Adaptive threshold: Welford's online variance tracks the running mean and variance of tag-centroid distances within each episode. The drift threshold is mean + k * sqrt(variance) (k = episode_drift_sigma, default 2.0), adapting to the project's natural tag-space movement scale.
• Revisitation: Before creating a new episode, the tracker checks the 50 most recent dormant episodes. If one is closer than the drift threshold, the old episode is revisited (visit count incremented, centroid EMA-blended) rather than duplicated. Episodes form a graph via transitions, not a timeline.
• Sync boundaries: Strong synchronicity events (effective_strength >= episode_sync_boundary_strength) force an episode boundary, marking the transition with the sync event ID.
• Transitions: Episode-to-episode edges record transition type: drift (gradual divergence), revisit (return to old topic), sync_boundary (forced by sync detection).
• Cross-episode bridges: Memory-pair edges (from the episodic tensor) that span different episodes are identifiable via the memory_episode_assignments junction table. Bridges between revisited episodes are the strongest structural signals.

Forest + Canopy (emergent, structural) — Two layers that organize memories into blocks:

• Forest (base layer): The co-occurrence optimizer compresses local neighborhoods — memories that share many tags cluster into blocks. The bridge optimizer preserves global traversability by maintaining edges between blocks that would otherwise be isolated. Raw memories stay fixed; the forest reorganizes maps, not terrain.

• Canopy (growth layer): Discovers emergent memory clusters from episodic tensor co-activation and bridge edges. When memories are repeatedly retrieved together, the canopy observes connected components in the episodic graph, scores them by co-activation strength minus tag-overlap inhibition, and records growth. The canopy never decays, prunes, or deletes — it only grows. Tag-overlap inhibition ensures the canopy surfaces structure that tags alone don't predict.

• Directional Canopy (inflow + outflow): Tracks retrieval causality — query tags are SOURCE, result memories are DESTINATION. Two independent layers:

  • Inflow canopy (memory-space): Clusters of result memories that are convergence attractors — they get pulled in by diverse, unrelated queries. Scored by the diversity of incoming query contexts: inflow_diversity = 1 - mean_pairwise_jaccard(query_tag_sets).
  • Outflow canopy (tag-space): Query tag neighborhoods that are distributors — they trigger retrievals reaching diverse, distant result clusters. Scored by diversity and reach of result neighborhoods.

  Both layers feed per-forest-block scores into the bridge optimizer's hub penalty. Genuine attractors (high inflow) and genuine distributors (high outflow) get partial rescue from degree-based penalization. Generic hubs with high degree but no directional signal get full penalty.

The canopy's Ramsey adjacency filter (canopy_ramsey_min_shared_primes) gates tag-pair shape generation via glyph lattice prime-slot overlap, connecting the emergent graph structure back to the number-theoretic lattice.

Cold Start

The system bootstraps from nothing:

t=0    All tables empty. PCA basis = random orthonormal scaffold.
       No seed data, no pre-trained weights.

t=1    First memories stored. Embeddings cached, tags created.
       PCA basis still random (< 20 embeddings for meaningful SVD).

t=N    Tensor self-seeds from ingest data:
       - co-occurrence: memory PMI (tags on same memories)
       - semiotic: memory conditional probability P(B|A)
       - glyph: structural resonance from Ramsey lattice
       Semantic channel remains dark (no retrievals yet).

t=N+k  Retrievals build co-retrieval edges and tag hit history.
       - semantic channel activates (co-retrieval gate satisfied)
       - co-occurrence and semiotic strengthened by retrieval PMI
       - query relevance filter focuses downstream analysis
       - temporal episodes begin tracking tag-space drift
       Synchronicity detection fires (rate-limited, 1/min).
       Events placed on lattice. All four channels operational.
       Strong sync events force episode boundaries.

Each instance starts with a unique random PCA basis — a random 2D plane through the embedding space — giving the lattice immediate angular diversity. Once 20+ real embeddings exist, SVD replaces the scaffold with a data-driven projection.

Commons Layer

Cross-project signal federation via a shared SQLite database. Multiple Chicory instances (each a separate project) emit store/retrieve signals that are processed into commons memories with dual tags:

• Project-namespaced: projectname:tagname — tracks which project produced the signal
• Shared: tagname — enables cross-project pattern discovery

The commons layer itself is a full Chicory instance — all existing layers (trends, synchronicity detection, lattice, tensor) operate on the signal-derived data. This means cross-project synchronicities emerge naturally when unrelated projects start touching similar themes.

Optional glyph-aware federation adds heat tracking and synchronicity detection across project boundaries.

# Enable in .env
CHICORY_COMMONS_ENABLED=true
CHICORY_COMMONS_PROJECT_ID=my-project  # auto-detected from git repo name if omitted

Project Structure

chicory/
  config.py                     Configuration (ChicoryConfig, 70+ parameters)
  exceptions.py                 Custom exception types
  api/
    app.py                      FastAPI REST server with session management
    sessions.py                 Per-user isolated database sessions
    user_db.py                  User authentication database
    static/                     Web demo frontend
  cli/
    app.py                      Typer CLI: chat, ingest, watch, status, migrate, dashboard
    chat.py                     Interactive chat session with LLM + memory
    commands.py                 Slash commands (/memories, /trends, /phase, /sync)
    commons.py                  Commons federation CLI subcommands
    conversation_log.py         Conversation history persistence
    display.py                  Rich terminal formatting
  db/
    engine.py                   SQLite with WAL mode, thread-safe RLock
    schema.py                   Schema v29, versioned migrations with idempotency
  ingest/
    parsers.py                  40+ file types (PDF, docx, code, markdown...)
    chunker.py                  Section/paragraph/sentence splitting
    ingestor.py                 File/directory ingestion with dedup (SHA256 content hash)
    document_ingestor.py        Full-text document ingestion with chunking
    code_summarizer.py          AST-based code structural summaries
    ignore.py                   .gitignore-style path filtering for ingestion
    watcher.py                  Real-time directory monitoring (watchdog)
  layer1/                       Memory store, embeddings, salience, tags, FAISS
    memory_store.py             Core memory CRUD with embedding and salience
    embedding_engine.py         Sentence-transformer embeddings + FAISS index
    tag_manager.py              Tag assignment, letter decomposition, normalization
    salience.py                 Multi-tier salience scoring
    vector_index.py             FAISS index management
    glyph_encoder.py            ByT5 character-level encoder for glyph lattice
    glyph_bridge.py             GPT-GU glyph symbol translation layer
    glyph_lexicon.py            Hardcoded concept↔symbol mappings (~80 glyphs)
    glyph_analyzer.py           Content scanning for concept words → glyph mapping
  layer2/                       Trend engine, retrieval tracker, time series
    trend_engine.py             Level, velocity, jerk, temperature per tag
    retrieval_tracker.py        Retrieval frequency and tag hit logging
    time_series.py              Time series snapshot storage
  layer3/                       Phase space, synchronicity, lattice, tensor, centroid, Poincaré
    phase_space.py              Temperature × retrieval frequency quadrant placement
    synchronicity_detector.py   Dormant reactivation, cross-domain bridges, semantic convergence
    synchronicity_engine.py     Prime Ramsey lattice, tensor, resonance computation
    centroid_subgraph.py        Retrieval-driven active reweighting (EMA centroids + edges)
    tag_space.py                Independent tag-graph retrieval path (lexical + fan-out + inward ratio)
    tensor_cache.py             In-memory cache for tag_relational_tensor (~74K rows, ~6MB)
    lateral_inhibition.py       Data-derived tag anti-association (tag_inhibition) + shunting helper
    directional_flow.py         Inflow (convergence attractor) + outflow (distributor) observers
    chain_anisotropy.py         Directional chain analysis in the tensor graph
    semiotic_graph.py           Semiotic convergence graph for tag-pair analysis
    cross_project_alignment.py  Cross-project signal alignment
    poincare.py                 Poincaré disk projection (exponential map, geodesics, Einstein midpoint)
    square_finder.py            Ramsey square detection in the tensor
  layer4/                       Episodic tensor, temporal episodes, forest, canopy, meta-analysis, feedback
    episodic_tensor.py          Sparse memory-to-memory edge cache (~48 neighbors/memory)
    temporal_episodes.py        Drift-detected, revisitable tag-space episode tracking
    forest.py                   Forest reorganizer (co-occurrence + bridge optimizers)
    cooccurrence_optimizer.py   Local neighborhood compression into blocks
    bridge_optimizer.py         Global traversability preservation between blocks
    canopy.py                   Emergent memory cluster growth from co-activation
    directional_canopy.py       Inflow (convergence attractor) + outflow (distributor) canopy layers
    adaptive_thresholds.py      EMA-based threshold tuning with burn-in
    meta_analyzer.py            Higher-order pattern detection across synchronicity events
    feedback.py                 User feedback integration
  llm/                          Multi-provider LLM abstraction
    base.py                     BaseLLMClient interface
    client.py                   Anthropic Claude client
    openai_client.py            OpenAI client
    null_client.py              No-op client (MCP-only / embedding-only mode)
    factory.py                  Auto-detection: Anthropic → OpenAI → Null
    prompts.py                  System prompt construction
    types.py                    Shared LLM types
  models/                       Pydantic models
    memory.py                   Memory, Embedding
    lattice.py                  LatticePosition, Resonance, PoincaréCoordinates
    synchronicity.py            SynchronicityEvent
    trend.py                    TrendSnapshot
    phase.py                    PhaseQuadrant
    meta_pattern.py             MetaPattern
    retrieval.py                RetrievalResult
    canopy.py                   CanopyShape, ScoreBundle, InflowScore, OutflowScore
  orchestrator/                 Central coordinator
    orchestrator.py             Ties all layers together, background detection thread
    tool_handlers.py            Tool dispatch for MCP and CLI
  mcp/
    server.py                   FastMCP server exposing 12 tools (stdio transport)
  migration/                    Model migration utilities
  dashboard/                    Optional Dash web UI (plotly + networkx)
tests/
  conftest.py                   MockEmbeddingEngine (deterministic, no GPU)
  test_integration/             Full-cycle tests
  test_layer3/                  Lattice, tensor, semiotic layer tests
  test_llm/                     LLM client tests
  test_glyph_analyzer.py        Glyph text analysis tests
examples/
  ramsey_network_demo.py        Self-contained demo (blank slate → self-organizing)

Runtime Loop

User sends message
  -> LLM generates tool calls
     -> store_memory(content, tags)
        -> [outside lock] glyph analysis, embedding (CPU-bound)
        -> [inside lock] assign tags, store memory + pre-computed embedding
        -> update tag centroids (EMA)
        -> forest reorganization (co-occurrence + bridge)
        -> episodic tensor: create candidate edges
        -> temporal episode: drift/revisit check, assign memory
        -> canopy observation (if block touched)
        -> invalidate tensor cache + PCA cache

     -> retrieve_memories(query)
        -> TagSpace lexical: tokenize query → match tag index → fan-out → lateral inhibition (lexical seeds) → score
        -> FAISS similarity search (single call, reused for tag filter)
        -> hub-filtered context tags (seed edge count pruning)
        -> candidate-scoped re-ranking (all via in-memory TensorCache):
           -> resonant memory lookup (tensor pair scores) ÷ (1 + β·H) from the top-5 context
           -> glyph association boost (candidate tags only) ÷ (1 + β·H) from the top-5 context
           -> semiotic convergence (batch signifies/signified_by)
        -> log retrieval
        -> query relevance filter:
           -> embed query, load centroids for result tags
           -> keep only tags above similarity_threshold
           -> filtered tags used for all downstream analysis
        -> record tag hits (filtered)
        -> centroid sub-graph reweighting (filtered):
           -> record co-retrieval edges (EMA)
           -> compute incoming association strengths (C @ C.T)
           -> rank, invert, scale by parallelness
           -> add/subtract to tensor + resonance strengths
        -> activate episodic edges for co-retrieved memory pairs
        -> temporal episode: drift/revisit check, assign retrieved memories
        -> forest reorganization + canopy observation
        -> directional canopy observation:
           -> compute query-side tags (centroid similarity to query embedding)
           -> record directional_retrieval_context (query_tag_hash)
           -> inflow observer: score result cluster convergence diversity
           -> outflow observer: score query tag set distribution reach
           -> update directional_block_scores for bridge optimizer
        -> [background thread]:
           -> update salience on access
           -> record trend events (filtered)
           -> reinforce synchronicity events
           -> _maybe_run_sync_detection (rate-limited)
              -> check_for_synchronicities()
                 -> detect dormant reactivation (phase space anomaly)
                 -> detect cross-domain bridges (resonance-gated)
                 -> detect semantic convergence (embedding similarity, no shared tags)
              -> place_events_batch(new_events)
                 -> compute angle (PCA -> atan2)
                 -> compute prime slots (150 primes, up to 863)
                 -> project to Poincaré disk (if enabled)
                 -> _find_and_persist_resonances (O(n) against existing positions)
                    -> INSERT resonances
                    -> UPSERT tensor synchronicity_strength
              -> force_sync_boundary (if effective_strength >= threshold)
                 -> snapshot current episode, start new or revisit old
           -> _maybe_recompute_inhibition (2% growth gate, lock released while counting)
              -> unit×tag incidence → BH-selected anti-associations → tag_inhibition (diff-write)
              -> swap the TensorCache inhibitor index

Configuration

Key parameters in ChicoryConfig (70+ total). Any of them can be passed to load_config(**overrides); only the ones load_config maps to an environment variable (among them CHICORY_DB_PATH, CHICORY_LLM_PROVIDER, CHICORY_INHIBITION_ENABLED and CHICORY_TENSOR_INHIBITION_WEIGHT) can be set from the environment or .env:

Parameter	Default	Purpose
llm_provider	auto	LLM provider: auto, anthropic, openai, grok, gemini, null
llm_model	gpt-5.5	Generic LLM model override
grok_model	grok-4.20-reasoning	Default xAI Grok model
gemini_model	gemini-2.5-flash	Default Gemini model
embedding_model	all-MiniLM-L6-v2	Sentence-transformer model
embedding_dimension	384	Embedding vector dimension
lattice_primes	[2, 3, ..., 863]	150 primes defining lattice scales
lattice_min_resonance_primes	30	Minimum shared slots for resonance
tensor_cooccurrence_weight	0.5	PMI weight in recall scoring
tensor_synchronicity_weight	0.3	Lattice weight in recall scoring
tensor_semantic_weight	0.2	Cosine weight in recall scoring
tensor_semiotic_weight	0.15	Conditional probability weight
tensor_inhibition_weight	2.0	Lateral inhibition gain β: partner activation ÷ (1 + β·H)
inhibition_enabled	true	Compute and apply data-derived lateral inhibition
inhibition_fdr	0.05	Benjamini-Hochberg false-discovery rate; evidence floor E > −ln(fdr)
inhibition_max_pairs	200000	Memory safety valve on stored pairs (strongest kept)

Tests

pytest tests/ -v                                    # Full suite
pytest tests/test_layer3/ -v                        # Lattice, tensor, centroid tests
pytest tests/test_integration/ -v                   # Full-cycle integration tests
pytest tests/test_llm/ -v                           # LLM client tests
pytest tests/test_glyph_analyzer.py -v              # Glyph analysis tests

Dependencies

Runtime: pydantic, sentence-transformers, numpy, faiss-cpu, scipy, typer, rich, python-dotenv, pymupdf, python-docx, watchdog

Optional: anthropic (Anthropic LLM), openai (OpenAI LLM), mcp (MCP server), fastapi + uvicorn (REST API), dash + plotly + networkx (dashboard), transformers + torch (glyph ByT5 encoder), chicory-commons-man (federation)

Dev: pytest, pytest-asyncio

Demo only: numpy (mock embeddings, no model downloads)

Database

SQLite with WAL mode. Schema v38, versioned migrations with idempotency checks. Thread-safe via threading.RLock on all execute/executemany calls.

Librarian (schema v29–v38)

Start here: librarian_desk. Everything else in this layer is pull — findings, commissions, FAQ drafts and custody flags accumulate quietly and are only ever seen by someone who already knew to look. The desk report is the one surface that speaks first: what changed since you were last here, what needs your judgment, what was learned, and the state of the building, in a single sub-second call. Reading it never acknowledges — the "since" window advances only on an explicit ack, so a report you glanced at and didn't act on comes back.

Custody layer over the store (chicory/layer4/librarian.py), wired through the orchestrator:

• Write gate — every store_memory embedding is scored against tag centroids (claimed tags first, best global match as fallback). Below librarian_gate_threshold the write is diverted to the quarantine ingestion tier and skips all network side effects, so a suspect memory cannot reshape the centroids it failed against. Ships in shadow mode (librarian_gate_enforce=False): verdicts are logged to write_gate_log for threshold calibration but nothing is diverted. Quarantined memories are excluded from FAISS, tag/semantic/hybrid retrieval, and episodic structure until released.
• Linear sweep — a cursor-driven patrol (memories.swept_at) that re-visits the least-recently-swept batch on an interval (background phase, librarian_sweep_interval_minutes). Mechanical checks: missing embeddings, near-duplicates, tag drift. Findings land in sweep_findings.
• Subject roster — connected components over centroid_edges (temporal/letter tags excluded), persisted to subjects with weighted centroids and mechanical names. Vanished subjects are retired, never deleted.
• Provenance — memories.author / memories.origin byline columns, accepted by store_memory and surfaced (with tier) in every retrieval result.
• Custody status (v30) — every memory carries a custody_status (unswept/clean/flagged/quarantined/certified); transitions are owned (gate/sweep/judge), journaled to custody_events, and surfaced in retrieval results alongside tier and author. A clean sweep pass never strips certification; only new findings demote.
• Demand ledger (v30) — every retrieval initiation lands in query_history; a background pass EMA-clusters them (query_clusters). Recurring well-served questions become FAQ candidates; recurring badly-served questions auto-file commissions.
• FAQ desk (v30) — faq_entries are drafted mechanically from demand but served only after judgment (approve), and serving re-verifies the supporting memories (broken entries go stale, never lie). The fast path answers known questions in one lookup.
• Specialist closure (v36) — work descends through layers (question → consult → shelf → commission → shift → report), and closure walks back up in , settling each layer it passed through: supersede the claims the report overturned, mark dependent FAQ answers stale, resolve findings it explains, close the ticket, and surface the change on the desk as — news distinct from . Supersession is (its own table, not the tensor's prunable supersession channel — a judgment that a claim was refuted must not age out), (the old memory keeps its byline and place, and from then on in every retrieval path), and is : by default, because a digest that got one section wrong is not thereby worthless. .

chicory status          # Show memory count, tag count, tensor state
chicory migrate         # Run model migration with burn-in period
chicory reembed         # Re-embed all memories with new model