czip — infinite context, without an infinite context window

October 6 update: optional Hermes context engine and job/source search now archives removed messages before compaction, preserves repeated requests and continuation aliases, and provides exact paged reads plus cross-session job search. czip find, find-index and opt-in find-drive use a local, evidence-bearing SQLite index with Turkish folding. Windows and POSIX launchers are included; activation is explicit and does not restart an existing service.

Optional local EmbeddingGemma 2 search combines semantic candidates with keyword results, preserves exact case filters and source hashes, and retains keyword search when the resident model service is unavailable.

🌍 English (primary) · Türkçe

When you carry a long agent session into a new chat, the entire history reloads into context. A 4 MB session = a 4 MB token bill.

czip compresses an agent's full history into a persistent, searchable pack (HKP1) — and the new session never unpacks it. It loads a ~1,500-token map, then retrieves only the message ranges it actually needs.

Headline

93,417 → 1,561 tokens. ~60× lower context-loading cost. Measured on a real 3,421-message working session (tiktoken o200k_base).

czip does not make the model's native context window infinite. It keeps long history persistent and searchable, and retrieves only the relevant parts when needed — practically unbounded retrievable history.

Finite context. Persistent memory.

See the memory claim tested

Czip Continuity Gauntlet is an open, reproducible benchmark for a local model working through one long engineering conversation. The same model answers with Czip retrieval, a rolling summary, or only its recent messages. The eight checks cover old facts, changed decisions, revoked permissions, reopened work, two-hop evidence, tool outputs, and honest abstention. Every run records exact answers, source indices, retrieval traces, token use when available, and dataset/engine hashes.

Try it with your own OpenAI-compatible local model:

git clone https://github.com/Jilazem/czip-continuity-gauntlet.git
cd czip-continuity-gauntlet
python bench.py generate --seed 20260930 --filler-per-gap 60 --output data/public-seed.json
# Run with --czip-source pointing to your separate Czip checkout; see benchmark README.

The benchmark tests the model plus memory workflow. Czip does not enlarge the model's native context window, and other retrieval systems can be tested on the same tasks.

In the 30 September 2026 local GLM run, Czip, rolling summary, and full history each scored 8/8 grounded answers on 615 messages; the last-24-message control scored 1/8. Czip used 71,243 prompt tokens across those questions versus 369,536 for full history and 88,171 for summary including its preparation calls. On two additional 4,015-message seeds, Czip scored 16/16 and the recent-message control 2/16. These are synthetic runs of one model and one task template; summary and full history were not tested at the larger size. The model's native token count for those larger histories was not available.

What's new (October 2026)

• Task store (czip promt / czip plan) — turn a raw task into a runnable prompt, get it approved, and save it as a tiny .hkp pack. Next session: /czip_promt oku <kid> brings it back for free. CLI: hkp.py gorev kaydet|oku|listele (gorev.py). Slash-command recipes live in commands/.
• Auto mode (czip-auto) — plugins-extra/czip-auto packs long, finished sessions in the background when a new session starts (every 30 min, CZIP_AUTO_ARALIK). Engine: czip_auto.py; thresholds via /czip_esik.
• czip-promt Hermes plugin (plugins-extra/czip-promt) — always-on context + gateway rewrite so czip promt <task> works in every channel, Telegram included. When a specialist/profile is assigned work, the task definition is saved first and the assignment body is just czip promt oku <kid>.
• KVKK/PII layer (kvkk.py) — personal data in a pack is tokenised and encrypted; external decision gates ask first.
• Search & merge upgrades — hybrid search index (depo.py), safer session merge (birlestir.py), ccd_dokum.py for Claude Code transcripts, local decision gate (laya_kapi.py, optional; set CZIP_KARAR_MOTORU).
• Paths are overridable: CZIP_HKP, CZIP_PYTHON, CZIP_PAKET_DIZIN, CZIP_AUTO_SCRIPT, HERMES_HOME.

v3 — the read cost was the real bottleneck

Compression ratio turned out to be the wrong metric: the pack file never enters context at all. The cost is what the reading agent pulls out. Measured on a real 3,421-message session:

Read path	Tokens
czip read (full index)	93,417
czip map (RAG map)	1,561
+ targeted czip search "<query>"	+634

60× cheaper. map does not dump the transcript — it returns a work map: role counts, tool histogram, user requests, recent messages, and a search instruction. The reading agent then pulls exact ranges on demand: search → range.

Two ideas were measured and rejected with data:

• Codec tuning. LZMA2 pb=0 lc=4 dict=256MB gained only 1.2%. bz2 was 27% worse. The current preset=9|EXTREME is already at the limit.
• Instruction language. Same instruction: Turkish 39 tokens, English 34, Chinese 35 — and the instruction text is 0.1% of total cost. The win is not in language, it is in JSON ceremony: rendering recent messages as A> text instead of JSON objects gained 47%.

v3 — duplicate elimination

Tool outputs are 71% of a typical pack, and 78% of them are exact duplicates (8,571 outputs → 1,854 unique). Second copies are replaced with [SAME-#N] markers.

12,217,023 → 479,724 B   (previously 562,416)   ratio 21.7x → 25.5x

Across five real sessions: 39.8 MB → 1.44 MB (15–35×), 2,590 duplicates eliminated.

v3 — merging (czip merge)

Detects sessions doing the same job and consolidates them into one pack; optionally archives the sources (close + archive; messages are never deleted, czip undo restores them).

The decision is made by Jev — and it actually matters: local similarity wanted to merge two unrelated case files at 0.75, Jev scored it 0.13 and rejected it; a true copy was approved at 0.97. Scheduled-task boilerplate is filtered out (41 false matches → 20).

czip merge               # read-only candidate scan
czip merge auto --jev    # merge + archive sources
czip undo <file>         # restore

czip pack also warns after packing when "another session did this job too" (the decision again belongs to Jev).

Numbers (from real sessions, evidence-backed)

Session	Raw	Pack	Ratio
Game dev (1,117 msgs, 31 long tool outputs)	4.27 MB	237 KB	18.0×
Code / research (210 msgs)	997 KB	119 KB	8.4×
New session's context entry	instead of 4.27 MB	~40 KB index + recent msgs

Why never unpack?

Traditional compaction either rewrites the whole history into the new session (expensive) or summarizes it (lossy). czip does neither: with the map + on-demand range model, the new session sees every one of 1,117 messages as a single index line and fetches any message by number. Dictionary tokens (␟3␞) look nothing like natural text — no collision risk — and the round-trip is bit-exact (test suite: 210/210 messages, bit-for-bit).

Install

git clone https://github.com/Jilazem/Czip && cd Czip
./install.sh

Installs: czip CLI (~/.local/bin) + Hermes plugin (/czip, /cunzip, /cmap, /csearch slash commands) + skill bridge + a 7-tool MCP server. After restarting Hermes, the slash commands appear as real commands in new sessions.

Commands

English aliases work everywhere (pack = Turkish paketle, etc.); Turkish remains the native naming:

czip pack <session_id|last|longest> [--full] [--jev]  → pack + 6-char id
czip read <id|last>           → WITHOUT UNPACKING: index + recent + status
czip map <id|last>            → RAG map (~1.5k tokens; the cheap read)
czip search <id|last> "query" → search inside a pack, find message indices
czip range <id|last> <a-b>    → exact messages, tokens resolved
czip merge [auto|<id1> <id2>] [--jev] [--days=7]
czip undo [<file>]            → restore archived sessions
czip list                     → registered packs: id | date | title

Carry-over flow in a new session:

czip map a37tvc               # which messages hold what
czip search a37tvc "edge case"
czip range a37tvc 1040-1060   # only the part you need

Features

• Tokenization — repeated blocks collapse into dictionary tokens; raw duplication never enters the pack (the secret behind 18×).
• Lossless mode — --full: every byte comes back (8.4×).
• State detection — read reports awaiting_user_reply plus the last message's role, so the new session resumes exactly where the old one stopped.
• Privacy — state.db is always opened via a READ-ONLY URI; packs are written only under ~/.hermes/session-packs; nothing leaves the machine; the engine has no delete/corrupt capability.
• Optional Jev gate — --jev asks an external LLM (typesafe.ai System-One) in batches whether tool outputs above 1,200 B are needed; outputs containing error hints are never questioned. Off by default — it sends sample content out, so it requires an explicit flag + API key, and silently falls back to static behavior on network failure.

Components

File	Role
hkp.py	engine — pure stdlib (Python 3.9+, 0 deps): pack/read/search, CLI, state.db RO reader
czip	terminal entry point (one-line wrapper)
plugin/	Hermes plugin — /czip, /cunzip slash commands
mcp_server.py	7-tool MCP server (compress/open/guide/messages/…)
skills/	skill bridge — commands work even in old processes
scripts/kapak.py	Pillow script that renders the terminal cover
tests/	synthetic round-trip + CLI + range tests (4/4)

Format: HKP1

"HKP1" | meta_len (4B BE) | data_len (4B BE) | LZMA-9e(meta) | LZMA-9e(body)

Meta: title, dictionary, counters, format version, packing info. Body: message record list (role, content, tool calls, reasoning, timestamp) — resolved into readable text on demand.

Tests

python3 tests/test_roundtrip.py   # 4/4: pack→read→range + stats

Limits

• smart mode truncates tool outputs to 2000+2000 B head/tail slices (and reports it); use --full for a bit-exact archive.
• Pack files are plain-text LZMA — no encryption. Do not expose packs from sensitive sessions to anyone with disk access.
• The state.db schema varies with Hermes versions; the engine skips missing columns via PRAGMA.
• --jev is an external service (off by default); avoid it under data-protection policies.

Benchmarks

• Super-Context Proof (2026-09-29) — a 748,800-token corpus recalled at 100% (80/80) with 112-token prompts via czip RAG; blind control 0% (0/40); 2-hop chains 30%. Chart: PNG.
• Continuity Gauntlet — a reproducible, source-grounded comparison against rolling-summary, recent-message, and full-history baselines using the same local model. Run reports and raw traces include the development failure and final protocol.

License

• Individual / non-commercial use: free and open.
• Commercial use: separate license required. Details: LICENSE

Versions before 2026-09-20 were released under MIT; those versions remain MIT-licensed.