Leibniz Legible

Live at https://leibnizlegible.com (since 2026-09-21).

An open access layer for the digitized Leibniz Nachlass — every one of the 236,795 page records held by the Gottfried Wilhelm Leibniz Bibliothek (GWLB), Hannover, machine-transcribed with per-line confidence and provenance, typo-tolerantly searchable, and browsable in an open IIIF viewer cross-linked to the scholarly catalogue.

This is machine output with honest labels — Vorausedition-grade, explicitly subordinate to the Akademie-Ausgabe. It is never "an edition". The edition is about half way through the volumes it plans (68 by 2023, completion expected around 2055), and one of its editors estimates that three quarters of Leibniz's papers have never been published anywhere (Kliege-Biller, Münster, 2023); making all of it legible and findable is the point.

Scope. The site covers the scans the GWLB has published of its Leibniz collections, not all of Leibniz's papers: much is not yet digitised, at the GWLB and elsewhere, and other institutions hold manuscripts too, the Niedersächsisches Landesarchiv Hannover among them; the Arbeitskatalog der Leibniz-Edition records them all, this site holds the whole Arbeitskatalog from the Arbeitsstelle's own export of 19 October 2026 (70,405 records, not the catalogue's website; refreshed with each export: leibniz catalog import-access), and every work page links to the catalogue's own search by shelfmark. Where Leibniz's papers are held, institution by institution, is on the holdings page and in reports/holdings.md.

• SPECS.md — the project's law (mission, deliverables, architecture, legal rails).
• PROMPTS.md — the phase-by-phase build plan.
• STATUS.md — living state; read it first.

Quickstart

Requires uv. Python 3.12 is provisioned by uv (the system interpreter is not used).

uv sync --python 3.12      # create the venv (CPython 3.12) and install everything
uv run leibniz --version   # -> leibniz 0.1.0
uv run leibniz info

uv run ruff check .        # lint
uv run ruff format --check . # style
uv run pytest              # tests (all offline)

Harvest (Phase A1)

# Set a contact for the polite crawler (SPECS §7.4); copy .env.example → .env.
export LEIBNIZ_CONTACT_EMAIL="[email protected]"

uv run leibniz harvest oai            # OAI-PMH → works (all 5 Leibniz sets; ~15 min, ≤1 req/s)
uv run leibniz harvest census         # write reports/census.md (the first real page count)
uv run leibniz harvest manifests --set leibniz-rekonstruktionen  # IIIF manifests → pages (a slice)

Both harvest commands are cache-first and resumable (raw XML/JSON under data/oai/, data/manifests/); re-running re-parses from cache without re-fetching. See reports/crawl-posture.md for the crawl-etiquette baseline.

Image cache (Phase A2)

uv run leibniz images pages          # derive all pages' delivery URLs from cached METS (offline)
uv run leibniz images fetch --work 00067974 --work DE-611-HS-854976  # pull a dev slice
uv run leibniz images verify --deep  # re-checksum the cache, report gaps
uv run leibniz images stats          # counts/bytes/dimensions → reports/census.md
uv run leibniz images duplicates     # sheet-sides registered under two folio labels → reports/duplicates.md
uv run leibniz images twins --out data/search.twins.json  # the same from the store alone, spread or fold

Sheet-sides are registered twice. The GWLB photographs an unfolded sheet as one image — two folio pages side by side — and registers that image under both folio labels (outer side 1r + 2v, inner side 1v + 2r), each label with its own copy of the file: the copies differ only in the caption stamped in a corner, so their checksums differ and their sizes almost agree. This happens in the IIIF (LH) works as well as in the static-JPEG ones. The corpus run read every such image once per label, so the page and line totals below count those repeats. Since 2026-10 the index build finds them from the store alone (same image size, file sizes within 0.1 %, a recto and a verso, confirmed by the lines lying in the same places; src/leibniz/images/twins.py): a two-page spread is indexed as its two halves, split at the fold (verso left, recto right), any other repeat on one page, and the site, the exports and the About page's figures follow it. The groups are written beside the index (search.twins.json, <store dir>/<index uid>.twins.json, or LEIBNIZ_TWINS_PATH); leibniz images twins writes the same file without an index build.

Caches one JPEG delivery derivative per page under data/images/ — resumable, checksummed, integrity-retried. The full-corpus pull (~365 GB) is an operator command; a --set/--work/--limit slice pulls a dev corpus. Images are never rehosted — this is an internal working store for the HTR pipeline.

Katalog crosswalk (Phase A3)

uv run leibniz catalog scrape --sample   # scrape a cross-set sample of Ritter-Katalog records
uv run leibniz catalog crosswalk         # match records → works (GWLB link + shelfmark)
uv run leibniz catalog report            # write reports/crosswalk.md
uv run leibniz catalog letters           # Bodemann's letters (correspSearch, CC BY 4.0) → letters.json beside the store

Joins the BBAW Ritter-Katalog (CC BY 4.0) to our works so every scan links to its scholarly record. No public API; the site caps results at 5000 rows, so the full scrape partitions into sub-cap slices (an operator job).

GT factory (Phase C2)

uv sync --extra gt                              # + pdfplumber (PDF text layers)
uv run leibniz catalog scrape --expired-volumes # every §70-expired volume's katalog records (~8 min)
uv run leibniz catalog crosswalk                # records → works
uv run leibniz align pieces                     # enumerate the §70 pieces localizable to canvases
uv run leibniz align ingest                     # fetch + extract each volume's reading text (cache-first)
uv run leibniz align edition-cache              # join to the katalog → data/gt/edition_cache.jsonl
uv run leibniz align gt-reset                   # a full re-mint: clear the factory's lines first (back up the store)
uv run leibniz align factory data/gt/edition_cache.jsonl --shard 1/6 --resume  # ×6 workers (needs the C1 HTR lines)
uv run leibniz align gt-report                  # writes reports/gt-factory.md
uv run leibniz align audit-sheet --images /path/to/image-cache   # 200 line strips → reports/gt-audit/gt-audit.html
uv run leibniz align audit-score reports/gt-audit/gt-audit-verdicts.csv  # precision per stratum → reports/gt-audit.md

The reading text of each §70-expired Akademie-Ausgabe volume is read from a free digital copy of the print (archive.org scans with their OCR layer, the GWLB repositorium PDFs, Potsdam's born-digital PDFs — align/volumes_sources.py records the channel and its terms per volume) by a layout-aware extractor (align/edition.py: running head, body vs. apparatus type, piece headings, margin numbers) — apparatus, commentary and introductions never enter it (SPECS §7.2). Minting itself runs where the C1 corpus store lives.

Kurrent track (K1, K2, K2b, K3)

uv sync --extra bench --extra gt --extra kurrent   # kraken + pdfplumber + transformers (TrOCR), in a side environment
uv run leibniz align kurrent-tolerance          # how much HTR noise the aligner survives → reports/kurrent/align-tolerance.md
uv run leibniz align kurrent-census             # the German pieces of the edition cache → reports/kurrent/census.md
uv run leibniz bench kurrent-smoke              # the candidate readers ranked on Dresden → reports/kurrent/bootstrap-candidates.md
uv run leibniz align kurrent-pilot --images /path/to/image-cache   # readers on Leibniz's German, a factory dry run → reports/kurrent/pilot.md
uv run leibniz align letterpool-fetch           # the GWLB letter pool's checked volumes (1708–1716) → data/external/letterpool/, the registry in reports/kurrent/letterpool-sources.json
uv run leibniz align letterpool-parse           # the PDFs into letters (heading, witness, reading text) → data/kurrent/letterpool/, counts in reports/kurrent/letterpool-parse.md
uv run leibniz align letterpool-set             # the German letters placed on the store's pages, the test set drawn and frozen → reports/kurrent/letterpool-testset.csv
uv run leibniz align letterpool-eval --read     # the readers on the frozen set (GPU, resumable), then the error rates → reports/kurrent/letterpool-eval.md
uv run leibniz align kurrent-read               # every line of the German pieces read by the Bern reader into data/kurrent/k2-readings/ (K2; resumable)
uv run leibniz align kurrent-mint               # the factory over those readings, the rule, the dry run → reports/kurrent/k2-mint.md; --write mints after a backup and a go
uv run leibniz bench train-set --sample 1000    # K3: the German mint selected, tagged, screened, split (the validation pieces frozen) and sized → reports/kurrent/k3-trainset.md
uv run leibniz bench train-set --workers 8      # … then cut into data/train/k3/ (image + .gt.txt, Kraken's format; the TrOCR loop reads the same files)
uv run leibniz bench train-trocr --pilot        # the memory settings timed on the Bern checkpoint (AdamW, Adafactor, frozen encoder, checkpointing, fp16)
uv run leibniz bench train-kraken --pilot --start   # one ketos epoch from the PHILIUMM checkpoint with the codec grown; --account reads it back and scores it
uv run leibniz bench k3-report pilots           # the two pilots against the week → reports/kurrent/k3-pilots.md

The German stratum has no ground truth of its own, so the Kurrent track measures before it trains: K1 ranked the public Kurrent readers and piloted them on Leibniz's German without ground truth; K2b built the honest test set from the GWLB Leibniz-Archiv's checked transcriptions of the correspondence of 1708–1716 (CC BY-NC 4.0: internal evaluation only, never on the site, in a release or in training — the texts stay under data/, the reports carry numbers) and measures every reader on it letter by letter, nothing filtered.

Search, viewer, IIIF (Phases D1/D2)

The viewer shell is stamped per route (title, description, canonical URL, share metadata) and, for /browse, /work/{id} and /page/{id}, carries a server-rendered summary — the whole index by shelfmark, the catalogue entries and page list, or the machine text — so search engines, answer engines and readers without JavaScript see the content; the app replaces it on boot. /sitemap.xml lists the index and every work, /llms.txt describes the API for agents, /robots.txt opens the JSON routes and keeps blind crawlers off the IIIF manifests. The shell names what it loads by content, computed at startup — the stylesheet and boot script by hash (?v=…), the ES modules as one set under /static/m/<build>/ — so a deploy reaches returning readers at once, and never as a mix of new modules and cached old ones.

uv sync --extra web                        # fastapi + uvicorn
uv run leibniz index build                 # page index → data/search.sqlite (SQLite FTS5)
uv run leibniz index query "calculemus"    # try it from the shell
uv run leibniz serve                       # http://127.0.0.1:8000 — search, /browse, /work/…, /page/…, /about
curl -OJ http://127.0.0.1:8000/api/works/00068642/text    # a work's text → leibniz-legible_00068642.txt
# production: typo-tolerant Meilisearch
docker compose up -d meilisearch
uv run leibniz index build --backend meili --meili-key "$MEILI_MASTER_KEY"
uv run leibniz serve --backend meili --host 0.0.0.0 --base-url https://your.host

The index folds text and query onto the aligner's early-modern comparison alphabet (u≡v, i≡j, ſ→s, diacritics, ligatures), so ut finds vt. A page must hold every word (match=any for one of them); titles, shelfmarks, AA references, folio labels and the titles of the catalogue pieces on a page are folded the same way, so AA VI,4 N. 109 or Praefatio find their pages. The pieces are those of the catalogue records the store holds: the whole Arbeitskatalog since its import from the Arbeitsstelle's export of 19 October 2026 (reports/crosswalk.md, reports/holdings.md). /api/lookup reads a query as a citation — a shelfmark with its folio, an Akademie-Ausgabe number, an id — and the search page offers where it points. Queries take two operators: "…" (or „…“, «…») for words that must stand exactly so and in that order, and a leading - to leave out pages with a word or a quoted phrase; both match exactly, so they cannot see past a misread word. The API (/api/search, /api/works/{id}, /api/pages/{id}, /api/stats) serves every line with its geometry, text, confidence, status and provenance; /api/pages/{id}/text and /api/works/{id}/text are the same transcription as a plain-text download (one line per recognised line under a # header carrying page URL, source image, model, run, confidence, licence and the wording rule; the work's pages streamed in canvas order, each under a ## Folio <label> — <page id> line; ?format=tsv for line_id, line_seq, conf, status, text rows) — the viewer links both: every page view has a Text block, the page's lines as one piece of selectable text with a Copy button and the download beside it. /api/records/{record_id}/text is the text of one catalogue piece — a letter, a draft, a treatise — across the folios its shelfmark names, placed on the scan by the C2 resolver (web/pieces.py), in the work export's layout under a header naming the record; the work page links it as "Text of this piece (Bl. 1–2)" under each record that can be placed, and a record that cannot be answers 404 with the reason. /api/works (no id) lists every work as a compact row together with the tree behind /browse (web/browse.py): shelfmark family → section → works, the manuscripts by LH section under the name the library's titles give it, the letter convolutes by correspondent — a name is shown only where the linked catalogue records name one that fits the alphabetical order of the LBr numbers — and the annotated books by number (, narrow it). Each work is also a IIIF Presentation 3 manifest () whose canvases reference W3C annotation pages () carrying the transcription — the interop deliverable D7. The viewer under is plain ES modules (no build step): OpenSeadragon on the GWLB's own Image API (static JPEG where no service exists) or on the project's image mirror when is set, a line-polygon overlay, status badges, confidence bands, provenance per line, EN/DE.

Since 2026-10: lines are numbered from 1 and citable (/page/{id}#L3; a Cite box with model, run and a permalink; ?run=N reads a page as run N left it). A scan the library registered under two folio labels is shown and indexed once — a two-page spread split at the fold — from the twins file the index build writes (LEIBNIZ_TWINS_PATH, else beside the index). /letters and /api/letters list Bodemann's letters by correspondent, date and place from letters.json (LEIBNIZ_LETTERS_PATH, else beside the store), and each letter convolute's page lists its own; /api/contents and the browse filter find the convolute that holds a piece by its title. Catalogue records link out to the Leibniz-Katalog, the cited volumes of the Akademie-Ausgabe, the piece's folio at the GWLB and, for letters of 1708–1716, the Leibniz-Archiv's transcriptions.

Releases (Phase D3)

uv sync --extra release                    # pyarrow
uv run leibniz release export              # data/release/leibniz-{inventory,transcriptions,gt}/
uv run leibniz release checklist           # the upload runbook (reports/release-checklist.md)

Each export ships Parquet (or --format jsonl), a MANIFEST.json with row counts and SHA-256s, and a dataset card with provenance, licence, attribution, error rates and the anti-contamination note. reports/tier1-final.md states the project against SPECS §3, criterion by criterion.

Deploying (a public host)

deploy/prepare-store.sh                                # desktop: compact, verified serving copy of the store
curl -fsSL https://raw.githubusercontent.com/marchofhares/leibnizlegible/main/deploy/install.sh | bash  # server: users, venv, Meilisearch, Caddy, units
leibniz index build --backend meili                    # server: one pass over the store; also writes the twins file
leibniz catalog letters --db /var/lib/leibniz-legible/inventory.sqlite  # server: letters.json beside the store
leibniz index bench --url https://your.host            # search p95 against SPECS §3.3 (500 ms)

uv run leibniz images thumbs                              # desktop: thumbnails for the mirror (Pillow, all cores)
rclone sync data/images r2:leibniz-images && rclone sync data/thumbs r2:leibniz-images/thumbs
uv run leibniz images check-mirror --base-url https://images.leibnizlegible.com  # HEADs a sample

deploy/README.md is the runbook: sizing, the store copy, install.sh, the search-only Meilisearch key, the index build, verification, measurement, the hardening that is on by default (per-client rate limit, security headers, CORS for IIIF consumers, read-only store, seven-day logs, robots.txt) and day-two operations. Dockerfile + deploy/docker-compose.prod.yml are the all-container alternative. leibniz serve takes every option from the environment too (deploy/env.example), runs --workers N, and answers /healthz for the process manager. Nothing on the server is precious: the store is a copy, the index a rebuild, and the images stay at the GWLB. The game Calculemus shares the host under its own subdomain and its own kit; deploy/README.md §13 covers what that touches here (a Caddy conf.d import, memory, backups) and the LEIBNIZ_CALCULEMUS_URL switch, off by default, that alone makes the About page link to it.

Where things live

Path	What
src/leibniz/	the package: one CLI (leibniz), one subpackage per pipeline stage
src/leibniz/db.py	canonical SQLite schema (SPECS §4.3) + typed helpers
src/leibniz/legal.py	§70/§71 copyright-expiry registry for AA reading text
src/leibniz/net.py	polite HTTP client — UA, ≤1 req/s per host, backoff (SPECS §7.4)
src/leibniz/harvest/	OAI-PMH + IIIF harvest → works/pages, and the corpus census
src/leibniz/images/	local delivery-derivative image cache (fetch/verify/stats), thumbnails + mirror check
src/leibniz/catalog/	Ritter-Katalog scraper + works crosswalk (shelfmark normaliser)
src/leibniz/htr/	benchmark harness (D5) + engine adapters (Kraken / VLM) — B1
src/leibniz/layout/	Kraken baseline segmentation of a page into line images
src/leibniz/align/	retro-alignment engine + the C2 GT factory (resolver, anchor-guided banded DP, strata, volume sources, reading-text extractor, ingestion)
src/leibniz/pipeline/	corpus segment/recognize batch pipeline + per-page seg-stats — C1
src/leibniz/search/	page index: folding, snippets, SQLite FTS5 + Meilisearch backends — D1
src/leibniz/web/	FastAPI JSON API, IIIF v3 manifests + annotations, the static viewer — D1/D2
src/leibniz/release/	Parquet/JSONL dataset exports + cards — D3
tests/	offline tests mirroring the package
data/	working store — git-ignored, never committed (see data/README.md)
reports/	committed reports (census, benchmarks, alignment yield)
deploy/	the deployment kit: runbook, install.sh, systemd units, Caddyfile, env files, production compose

Status

The v1 corpus run is complete (2026-09-11): 236,210 of 236,795 page records — 99.75% of the digitized Nachlass — are machine-recognised, 13.5M line records with per-line confidence and provenance (sheet-sides registered twice included; see the note on them above). The remainder is enumerated (569 skips with reasons; 16 pages behind broken GWLB redirects). Phases A0–A3, B1–B2, C1 are complete (STATUS.md has the detail). A1's census found 2,225 works / 236,795 page images; A2/A3 built the image cache (395.6 GB) and katalog crosswalk. B1 reproduced the PHILIUMM HTR CER (7.95 % vs claimed 8.33 %) and B2 green-lit the retro-aligner (98.8 % yield / 97.5 % precision on favourable material). C1 built — and has now run at corpus scale — the resumable segment/recognize pipeline (leibniz pipeline, sharded parallel workers + concurrent GPU recognition); C2 built the GT factory (leibniz align factory) and has now run its inputs at full scale: the §70 katalog sweep (17,162 piece citations, 11,595 localizable) and the reading-text extraction of 21 of 31 expired volumes from their free digital copies (6,188 pieces, 26.2M characters, 10,029 witnesses with text). The mint has run (2026-09-16): 297,424 open-bucket ground-truth lines, 5.9× the 50k target, in about two hours on six workers. Its precision was hand-audited on 199 of 200 sampled lines (2026-10-07; reports/gt-audit.md, numbers in reports/gt-audit/audit-summary.json): corpus-weighted 72.3 % as written, below the 95 % gate, and 92.2 % counting a line a letter or a word off at an end as usable. The patterns behind it are fixed in the factory (the scribe's line-end hyphen kept, minted text one line, a re-mint replacing one catalogue record's lines only) and land with the one re-mint before C3. Phase D is built on v1 (2026-09-16): the search index, the JSON API, IIIF Presentation 3 manifests with W3C annotations, the viewer, and the dataset exports with cards — the v1 public beta; the operator runs leibniz index build + leibniz serve on the corpus store. The deployment kit landed the same day (deploy/: runbook, systemd + Caddy, container stack; the app hardened for public traffic — rate limit, security headers, CORS, read-only store, /healthz, leibniz index bench); the site went live on 2026-09-21 (deploy/README.md is the runbook). Next is C3 (fine-tune v2, a gate; see the amended prompt), then C4 re-reads the corpus and swaps v2 in under a new run. Reports are on Zenodo: project statement 10.5281/zenodo.22782813, census , PHILIUMM reproduction , retro-aligned GT . See ; holds the 2026-09-16 strategy review.

Licensing (summary — see SPECS §7)

Code is Apache-2.0. GWLB scans are Public Domain Mark 1.0. By default the app loads them from the GWLB's own endpoints (SPECS §3.4); the public site serves its own mirror of the GWLB's delivery scans instead (LEIBNIZ_IMAGE_BASE_URL; an operator decision recorded in STATUS.md, Divergences), with every page linking to its original at the GWLB. Only §70/§71-expired AA reading text is ever redistributed; NC-licensed sources stay in a quarantined bucket excluded from CC BY releases and the UI. The transcriptions are CC BY 4.0, machine output of a model obtained under CC BY 4.0 (the PHILIUMM HTR model, published by its makers under CC BY-NC 4.0 since 2026-10-15, with their agreement that the transcriptions stay CC BY). The ground truth is two deposits by channel: open (CC BY 4.0) and NC-derived (CC BY-NC 4.0: the Potsdam Reihe IV PDFs, the GWLB repository's PDFs). A model fine-tuned from the PHILIUMM checkpoint is CC BY-NC 4.0; one with no NC input CC BY 4.0. The licence state of every third-party artifact and the licence of every output are in src/leibniz/legal.py (uv run leibniz legal licences, leibniz legal outputs; SPECS §7, amendment of 2026-10-21). Provenance is mandatory on every stored and exported line.