openodke

An open implementation of ODKE+ (Apple, arXiv:2509.04696) — independent, and not affiliated with Apple.

PyPI · Documentation · Changelog — pip install openodke

The verification layer between any extractor and your graph store. It doesn't matter what produced your triples: LangChain's LLMGraphTransformer, neo4j-graphrag, LangExtract, patterns over a table, or openodke's own extractor. openodke checks each fact against the text it came from, links its entities, counts the independent sources behind it and measures the result. Then it writes the graph to Neo4j, RDF or JSON Lines with the evidence attached.

Every fact carries the document, the character span, the source tier and the grounding verdict, so why is this edge in my graph? is a query, not an investigation.

Status: 1.0.0rc1, a release candidate; 0.2.1 is the last full release on PyPI. 1.0 is the verification layer (DECISIONS #24): the Validator runs on any extractor's triples (odke validate, with adapters for LangChain, LangExtract, neo4j-graphrag and a graph already in Neo4j), inside openodke's own pipeline (odke run), and on a graph openodke did not build (odke ground); the Evaluator (odke eval, odke bench) measures it.

On public documents the grounder removed 9–20% of the wrong triples each extractor wrote, for 16–37% of extraction's cost; on single sentences it cost precision (Benchmarks). No grounder has met its calibration bar yet, so by default what it would refuse is marked for review, not dropped. See ROADMAP.md.

Quickstart

From a clone, with no key, no network and no database:

git clone https://github.com/deepskandpal/odke && cd odke
pip install -e ".[yaml]"
odke run examples/e2e/odke.yaml

That runs a small invented corpus through all thirteen stages on recorded model responses and writes the graph to examples/e2e/out/. The same run from Python:

from openodke import HybridExtractor, LLMExtractor, Ontology, Pipeline, VerdictGate
from openodke.ground import LLMGrounder
from openodke.llm import RecordedClient, ReplayClient
from openodke.loaders import DirectoryLoader
from openodke.sinks import JsonlSink

ontology = Ontology.from_json("examples/e2e/ontology.json")
docs = list(DirectoryLoader().load("examples/e2e/corpus"))
# Recorded responses, so this runs with no key. Drop `client=` to call the models.
extract = ReplayClient("examples/e2e/recorded/extract.json")
ground = RecordedClient.from_fixture("examples/e2e/recorded/ground.json")
kg = Pipeline(
    ontology,
    HybridExtractor(LLMExtractor(client=extract), documents=docs),
    grounder=LLMGrounder(client=ground),  # a second model checks every cited span
    gate=VerdictGate(),  # marks what its span contradicts, below the grounder's bar
    sinks=[JsonlSink("out")],
).run(docs)
print(len(kg.facts), "facts,", kg.stats.get("marked", 0), "marked for review")

examples/e2e/ goes on into Neo4j and ends with the Cypher that shows provenance, a DIFFERENT link between two companies with one name, and a cardinality check.

Does it work?

On the example, odke eval ablation runs one config three ways against 36 labelled facts:

Configuration	Precision	Recall	Model calls
extraction alone	0.889	0.889	3
+ grounding	0.889	0.889	39
+ corroboration	0.944	0.944	39

This is a demonstration on hand-authored recorded responses, not a benchmark. The responses and the labels were written for the example, so the table shows what the harness reports, not how well any model does. On that fixture the grounder caught one of four wrong candidates, and the gate marked it for review rather than dropping it: the default grounder is below its calibration bar, so nothing is dropped on its word (refusing it would have given 0.914). Normalisation did the rest. The paper's 98.8% is neither reproduced nor claimed. What the example shows, and does not.

On public data, five extractors run with and without the layer, with ODKE+'s paper-mode grounder reading each whole document (Claude models, uncalibrated), not the Validator's default span prompt. On Re-DocRED's documents the grounder raised precision for all four model-based extractors, by 1.8 to 4.3 points, for one Haiku call per triple ($4.23 to $9.01 per 1,000 documents), with F1 unchanged within its interval. On Text2KGBench's single sentences it lowered precision for every extractor, by 0.7 to 2.2 points. The pattern extractor found nothing in prose.

The harness is the point: run the same command on a slice of your own corpus that you have labelled, and it tells you whether grounding earns its calls there.

Why this exists

Apple published ODKE+ (arXiv:2509.04696), a production system that extracts open-domain facts at scale and ingests them into a knowledge graph at 98.8% precision. Apple released no implementation, and nothing on PyPI, conda, GitLab or Hugging Face implements it. This is an independent implementation of that architecture, generalised from one company's internal graph into an SDK anyone can install.

The three ideas worth taking from that paper:

Idea	What it does	Why the alternatives lose
Ontology snippets	Prompts the model with a small, ranked, per-type schema fragment rather than the whole ontology	A 200-predicate schema does not fit in a useful prompt. Snippets keep prompt size flat as the schema grows
Grounding verification	A second, cheap model checks each candidate fact against its own evidence span and records supported, contradicted or not_found on the fact; a gate decides what is written	Extraction alone hallucinates. A yes/no check against a quoted span is cheap, and a verdict kept on the fact can be measured and re-gated later
Corroboration	Merges the same claim across sources, and resolves conflicts on freshness × trust × agreement	Real corpora disagree with themselves. Without it you write both answers and cannot say which to believe

Other libraries turn text into a graph — iText2KG, neo4j-graphrag's SimpleKGPipeline, LangChain's LLMGraphTransformer. They extract, and most can be handed a schema. None of them has a separate grounding pass, cross-source corroboration, or character-span provenance carried through to the store.

Thirteen stages, each one a Protocol

The pipeline is a superset: thirteen stages, each a typing.Protocol in openodke.stages with a pass-through default (DECISIONS #20). Take the subset your corpus needs, supply your own for any stage by writing one method, and leave the rest out. Nothing domain-specific is in the code; it arrives as an ontology, a config, or a stage of yours.

Stage	Built-in	Name in odke run	Left out
load	DirectoryLoader, TextLoader, MarkdownLoader, HtmlLoader, PdfLoader, DocxLoader, CsvLoader, TsvLoader, JsonLoader, JsonlLoader, ParquetLoader	directory (every suffix), text, markdown, html, pdf, docx, csv, …	directory
chunk	SentenceChunker — whole sentences, offsets intact	sentence	one chunk per document
route	yours: skip marketing, policy, narrative	—	extract everything
extract	TriplesExtractor (any extractor's triples), PatternExtractor, LLMExtractor, HybridExtractor	triples, pattern, llm, hybrid	required
ground	SpanGrounder (free), LLMGrounder	span, llm	verdict unchecked
normalise	ValueNormalizer — dates, numbers, units, name keys	value	unchanged
resolve	NativeResolver — blocking, identifiers, links	native	keys as given
corroborate	SignatureCorroborator	signature	every fact its own claim
score	EvidenceScorer — verdict × support × extractor confidence	evidence	evidence under odke run; pass-through in a hand-built Pipeline
gate	VerdictGate — refuses contradicted

Install

From PyPI:

pip install openodke                   # the base install
pip install "openodke[yaml]"           # + YAML configs
pip install "openodke[neo4j,yaml]"     # + the Neo4j sink
pip install "openodke[all]"            # everything

For unreleased changes on main: pip install "openodke @ git+https://github.com/deepskandpal/odke".

The base install is pydantic and typer, and talks to nothing: the data model, the ontology compiler, the text, Markdown, HTML and record loaders, the pattern extractor, the OpenAI-compatible model client, corroboration, evaluation, and the JSONL and Cypher-file sinks. Each extra is imported on first use, and a missing one is named in the error.

Extra	Adds	For
llm	litellm	Model providers that do not speak the OpenAI shape
neo4j	the Neo4j driver	Neo4jSink, Ontology.from_neo4j
yaml	PyYAML	YAML ontologies and run configs
parquet	pyarrow	ParquetLoader
pdf, docx	pypdf; python-docx	PdfLoader, DocxLoader
rdf	rdflib	RdfSink, Ontology.from_owl
networkx	networkx	NetworkXSink
docs	pypdf, python-docx	The document readers together
bench	NLTK	odke bench on Text2KGBench (its hallucination metrics)
otel	the OpenTelemetry API	Spans per job, stage and model call
all	all of the above

Python 3.11–3.14.

odke run

One file names the inputs, the ontology, the models, which implementation fills each stage, and the sink. Here the extractor is any other system's triples, read from a file, checked against the texts they cite and written to Neo4j:

ontology: ontology.json
inputs:
  - path: texts                     # the texts the triples' `doc` names
    loader: {use: directory}
models:
  ground: anthropic/claude-haiku-4-5-20251001
stages:
  extractor: {use: triples, path: triples.jsonl, extractor: my-extractor}
  grounder: llm
  normalizer: value
  corroborator: signature
  gate: verdict
  sink: {use: neo4j, uri_env: NEO4J_URI, password_env: NEO4J_PASSWORD}
  constrainer: neo4j
bootstrap: true

extractor: llm (or hybrid, patterns first) extracts with openodke's own reference extractor instead, on the extract model.

odke run config.yaml --dry-run     # load, extract and ground; print what would be written
odke run config.yaml

A stage of your own is package.module:Name with its options alongside. A dry run opens no sink, so it needs no database. Every stage's own counts — verdicts, rejections, conflicts, refusals, cost — go into the graph's stats, and a document's id is its path, so a label or a query can name it. examples/run.yaml comments every key.

Evaluate against your own labels

openodke ships the formats and the arithmetic for scoring every stage against your own labelled dataset; odke bench runs the public ones (Benchmarks).

odke eval extract --describe                           # what to label
odke eval extract --labels gold.jsonl --predictions out/facts.jsonl
odke eval ablation --config config.yaml --labels gold.jsonl
odke eval spans --facts out/                           # no labels at all

There is an evaluator for routing, extraction, grounding, resolution, scoring (Brier, reliability, ECE) and validation, and openodke.eval.CostMeter measures tokens and USD per stage without touching a stage. One evaluator needs no labels: odke eval spans reports cited span width split by grounding verdict, which scores the too-narrow-citation error with no gold set. The fixtures in the test suite exercise the arithmetic and are not a benchmark.

When the platform does a stage itself

Some stores already resolve, prune or constrain after the write. A sink declares that with a PlatformProfile; a stage configured on both sides raises one DoubleStageWarning and still runs, because the two passes are not the same pass (DECISIONS #21). To hand a stage to the platform, pass Delegated(to="neo4j-graphrag:FuzzyMatchResolver") — or {use: delegated, to: ...} in a config — and openodke stamps who did it wherever the data model has room, so the platform's work can be read back and scored with the same evaluator.

Models and providers

Nothing in this library imports a provider SDK or names a vendor. Every model call goes through one LLMClient protocol, and which client serves a model string is decided by a small, inspectable routing rule you can override.

from openodke.llm import ModelRoles, ModelSpec

ModelRoles()  # Claude by default: Sonnet extracts, Haiku grounds
ModelRoles.single("ollama/llama3.1")  # entirely local — no extras, no key, no network
ModelRoles(
    extract=ModelSpec(model="anthropic/claude-sonnet-5"),
    ground=ModelSpec(model="ollama/qwen2.5:3b"),
)

Two models, not one, by default. Extraction wants a capable model; grounding asks thousands of yes/no questions and wants a small one. Anything speaking the OpenAI chat shape — Ollama, vLLM, LM Studio, llama.cpp, OpenRouter, Groq, a gateway — works on the base install; everything else goes through litellm with [llm]; openodke.llm.register("mycorp", factory) routes a provider through your own client. ReplayClient, RecordedClient and ScriptedClient ship in the package, so your own stages are testable offline too.

odke models                                  # every provider, its key variable, whether it is set
odke run odke.yaml --model openai/gpt-5.5    # override the config's models for one run

odke models runs on the base install with no key and no network, and never prints a value: keys are read from the environment and stored nowhere. See Models and providers.

Ontologies

from openodke import Ontology

ontology = Ontology.from_json("examples/e2e/ontology.json")  # or from_owl, from_neo4j
print(ontology.validate())  # diagnostics with dotted paths, never an exception
print(ontology.snippet("Company").render())  # exactly what the model is prompted with

odke ontology validate schema.json
odke ontology diff old.json new.json --fail-on-breaking
odke ontology snippet schema.json Company

A bad extraction is usually a bad snippet, and snippet shows it before a token is spent. The ontology also marks which qualifiers bear identity (percentile) and which reconcile (start_time), and that decides what counts as one claim.

Honest limits

• The fixtures and the example are not benchmarks. Their model responses are hand-authored. The numbers measured with real models are on Benchmarks.
• Live Neo4j needs 5.7 or later. The constraint bootstrap uses relationship uniqueness constraints; Community edition is enough.
• Structured facts are grounded against their own cell. The cell Leeds cannot say whose head office it is, so a careful grounder answers not_found and the call is spent for little. The default gate keeps not_found for this reason.
• The default grounder reads the cited span. That is what keeps it cheap, and a span that leaves out the subject cannot support the claim. Widen and retry (widen, off by default) asks again against the sentence, and context: document reads the whole document, as paper mode does.
• Resolution never merges a stored node on names. Only a shared identifier re-keys onto one; a name match is a SIMILAR link for someone to act on. Asked to (normalize_batch=True, off by default), it makes a batch's own look-alikes that nothing in their names or context keeps apart one entity before anything is written (DECISIONS #43).
• Inference drafts; it never decides. odke ontology infer proposes a small schema for a person to review and freeze, and nothing infers implicitly (DECISIONS #8).

Documentation

• openodke.dev — the documentation site: concepts, the Validator and its inputs, loading documents, grounding, resolution and corroboration, stores, odke run, evaluation, labels and benchmarks, models, the ontology and the reference extractor
• examples/ — the end-to-end example and the commented run config
• ROADMAP.md — milestones, what each one delivers, and the estimate
• CHANGELOG.md — what landed, milestone by milestone
• DECISIONS.md — the design calls and why they went that way
• CONTRIBUTING.md — ./scripts/verify.sh is the whole check
• NOTICE — the relationship to the ODKE+ paper

Relationship to the paper

This is an independent implementation of a published architecture. It is not affiliated with or endorsed by Apple Inc., uses no Apple code, data or models, and was written from the paper alone. All credit for the architecture belongs to Khorshidi et al. See NOTICE.

It departs from the paper where the paper is specific to its deployment. ODKE+ decides what to refresh by watching Wikipedia edits and retrieves its own evidence; here those two stages are optional protocols, Initiator and Retriever, because an SDK is usually handed its documents.

License

Apache-2.0.