Fact Knowledge Layer

Pulls checkable facts out of PDFs, keeps a pointer back to the exact span each one came from, and then figures out how facts from different documents relate. Do they agree? Do they actually conflict? Or do they only look like they conflict because one figure is for FY2023 and the other for FY2022, or one is consolidated and the other standalone?

Storing the facts is the easy half. The half worth building is deciding whether two numbers that disagree are actually a problem, and being able to show a reviewer why.

---

Setup and Run Instructions

Python 3.11+. Nothing else to install or stand up, no database.

git clone <this repo>
cd SuperJoin
./run.sh

run.sh makes a virtualenv, installs deps, generates the sample PDFs, and serves the UI on http://127.0.0.1:8000. Upload PDFs there and poke around.

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
cp .env.example .env                       # optional: add your API key
.venv/bin/python scripts/make_sample_pdfs.py
.venv/bin/python -m uvicorn factlayer.api:app --port 8000

You don't need an API key

Without a key it falls back to a rule-based extractor and rule-based reconciliation. That path is complete: all four required cases come out of the sample documents, and the full test suite passes. It just finds fewer facts, and it's especially bad at non-numeric ones.

If you want the LLM extractor, drop a key in .env:

FACTLAYER_LLM=gemini
GEMINI_API_KEY=your-key-here

FACTLAYER_LLM also takes anthropic, openai, groq, grok or none. Groq (gsk_... keys, open models over the OpenAI protocol) and xAI's Grok (xai-... keys) are two different services with confusingly similar names, so they have separate settings. Keys are only ever read from the environment, and .env is gitignored.

Running it as a public service

vercel.json deploys it as a Python function with the UI served statically. Two things behave differently there, both controlled by FACTLAYER_SESSIONS:

Every visitor gets their own layer. Uploaded PDFs are somebody's private documents, and the whole point of the system is that a new document gets compared against everything already in the store. One shared store on a public URL would show visitor A's documents to visitor B, and report contradictions between two strangers' files. So each visitor gets their own SQLite file and upload directory keyed by a cookie (see factlayer/session.py). Leave the variable unset and you get the original single shared store, which is what local runs and the tests use.

New sessions come pre-seeded. The sample documents get ingested once per process and each new session starts as a copy of that, so somebody landing on the URL for the first time sees the four cases instead of an empty page. The seed is built by actually running the pipeline (factlayer/seed.py) rather than shipping a prebuilt database, so it can't quietly drift away from what the extractor does now.

Storage lives under /tmp, which is per-instance and ephemeral. Uploads outlive the visitor's session but not the instance. That's fine for a demo. The real fix for durable multi-user storage is a proper database per account, not a shared SQLite file, and I'd rather say that than pretend /tmp is a solution.

To use the LLM extractor on a deployment, set FACTLAYER_LLM and the matching key in the project's environment variables. Nothing is read out of the repo.

Command line

.venv/bin/python -m factlayer.cli ingest 'data/samples/*.pdf'
.venv/bin/python -m factlayer.cli cases        # the four required cases
.venv/bin/python -m factlayer.cli facts
.venv/bin/python -m factlayer.cli status

Tests

.venv/bin/python -m pytest tests/ -q          # 53 tests, no API key needed

API

Endpoint	What it does
POST /api/documents	Upload a PDF (multipart). Ingests and links it.
GET /api/cases	The four required cases, as queries over the layer.
GET /api/relations?type=contradicts	Relations, filter by type or document.
GET /api/clusters	Facts grouped by the claim they're making.
GET /api/facts	Facts, filter by document/subject/attribute.
GET /api/evidence/{fact_id}	The fact's quote, in its surrounding page text.
GET /api/schema	The attribute/subject vocabulary learned so far.
GET /api/issues	Extraction and reasoning failures that got logged.

Interactive docs at /docs.

---aaaaaaaaaaa

Approach

The pipeline

PDF -> pages (text + char offsets + word boxes)
    -> passages (table header scale hints carried down)
    -> facts         <- LLM extractor, or rule extractor if there's no key
    -> grounding     <- every quote re-found in its page, or the fact is dropped
    -> normalisation <- values, units, scales, periods reduced to comparable keys
    -> candidates    <- blocked by canonical attribute, then scored
    -> relations     <- rules first, LLM only for what the rules can't settle

The fact model

A fact is thin on purpose:

subject | attribute | value | qualifiers | evidence | confidence

There's no fixed list of attributes. The documents decide what an attribute is. A SchemaRegistry folds near-duplicate predicates together as they arrive, so "total revenue", "Revenue from operations" and "total revenues" all end up in the same place. That means a new document can introduce a kind of fact nobody planned for and it'll still line up with what's already there. GET /api/schema dumps the vocabulary as it currently stands.

qualifiers is the field that does the most work: period, scope (consolidated or standalone), basis (audited or provisional), segment, as-of date. Nearly every apparent contradiction I hit in real documents turned out to be two facts with the same subject and attribute but different qualifiers.

Evidence isn't optional

Every fact carries a quote, and that quote gets re-located in the source page (exact match first, then whitespace and case-insensitive, then token-anchored). If the quote can't be found, the fact is thrown away rather than stored, and the drop is written to an issues log. This is the main thing standing between the system and a model inventing a citation, and there's a test for it.

How two facts get compared

Rules run first, because most of this is arithmetic and context that's already written down, and deterministic answers are reproducible and cost nothing. The important bit is the ordering: comparability gets checked before the values do.

1. Are the qualifiers different (period, scope, basis, segment, as-of)? Then the values differing is explained, not a conflict. Marked reconcilable, with the reason.
2. Do the normalised values agree within tolerance? Then corroborates, or rounding if they're close but not identical.
3. Same subject, same attribute, same stated context, values disagree? That's a contradicts.
4. Genuinely ambiguous? Hand it to the LLM, which gets both quotes and has to justify its answer against them.

Every relation records which method decided it, the confidence, and a plain English rationale. Nothing goes into the layer that can't explain itself.

The three decisions that mattered most

A contradiction needs the same subject. The quickest way to produce a confident wrong answer is to compare a parent company's number against its subsidiary's. So subject identity gets checked before any conflict is declared, and below the threshold the pair is reported as different entities instead. On the sample documents that one check killed three of the four "contradictions" the first version found.

Fuzzy matching must not paper over negation. "Profit before tax" and "profit after tax" are about 90% similar as strings, and merging them makes a company look like it's contradicting itself. There's a small table of polarity words (before/after, gross/net, basic/diluted, current/non-current, including/excluding, audited/provisional and so on) that keeps those attributes apart no matter how similar the text looks.

Missing context is not the same as matching context. If one document states a scope and the other doesn't, there's no way to show the two figures are measuring the same thing. So it gets escalated as uncertain instead of asserted as a conflict. "Can't tell" is a more useful answer than a confident wrong one.

Tables

Most of the real numbers live in tables, and a table row keeps its context up in the column header. Reading Total Revenue 1,240 1,050 without the Particulars FY2023 FY2022 above it is exactly how one row turns into two conflicting values for the same year. The table parser pairs each cell with its column's period, and a (Rs in crore) header multiplier propagates down into the rows below it.

Storage

SQLite, one file. A graph database would have looked more impressive and helped less. The hard part here is deciding what the edges mean, not traversing them. SQLite also means you can clone this and run it with nothing to provision.

Incremental by design

Ingesting a document extracts only that document's facts and then compares them against what's already in the layer. Existing facts and relations never get recomputed, so the cost of an ingest scales with the new document, not with the size of the corpus. Re-uploading the same bytes is caught by hash and does nothing. Candidate pairs are blocked by canonical attribute and capped per cluster, so one very popular attribute can't turn the whole thing into an O(n^2) mess.

AI tools used

Built with Claude Code (Claude Opus 5) as a pair programmer. It wrote most of the implementation. The design decisions above came out of running the pipeline on the sample documents, reading the false positives, and fixing what caused them. Gemini is the default backend at runtime for extraction and adjudication.

---

Limitations and Next Steps

What doesn't work yet, specifically:

Extraction

• The rule-based extractor is noticeably worse than the LLM path. It reads numeric facts fine, but it names some attributes badly (representing growth), occasionally mislabels a percentage, and finds almost no purely semantic facts at all (roles, obligations, relationships). It's there so the project runs without credentials, not as an equal option.
• No coreference resolution. Given "The audited consolidated figure subsequently reported was Rs 1,240 crore", the rule extractor can't tie "the figure" back to revenue two sentences earlier. So that fact gets a bad name and never links up with the provisional figure it should reconcile against. The LLM path gets it right. See Case 4 below, I left this one visible.
• No OCR, so scanned or image-only PDFs aren't handled. Text-layer PDFs only.
• Multi-level and merged-cell table headers aren't modelled. Only a single period header row is understood.

Reasoning

• Money in two different currencies comes back as related and never gets compared. There's no FX rate in the documents and making one up would be worse than declining.
• Period inference assumes an April to March fiscal year by default. It's configurable, but it isn't inferred from the document.
• The numeric agreement tolerance is one global number. A 0.5% gap means something very different for a headcount than for a revenue figure.

What I'd do next, in the order I'd actually do it

1. Highlight evidence on the rendered page. The bounding boxes are already computed and already come back from the API. The UI shows the quote but doesn't paint it onto the PDF yet.
2. Coreference resolution in the rule path, plus a second LLM pass that re-reads any fact whose attribute looks degenerate.
3. Embedding-based blocking instead of fuzzy string matching, so "number of employees" and "headcount" cluster without me hand-tuning a threshold.
4. Calibrate the confidences. Right now they're sensible constants, not calibrated against anything. I'd build a small gold set and tune to it.
5. Derived-fact reasoning: know that revenue minus expenses should equal profit, and flag documents whose own arithmetic doesn't add up.
6. OCR fallback for scanned documents.

---

Additional Notes

The four required cases

All four come out of the three sample documents with no API key, and you can see them in the UI's Four cases tab or via python -m factlayer.cli cases. None of it is hardcoded: /api/cases is just a set of queries over whatever happens to be in the layer.

1. Corroborated across documents, expressed differently The annual report says revenue was Rs 1,240 crore. The investor update says INR 12.4 billion. Different documents, different scale words, same number, same period. Normalisation takes both down to INR:1.24e+10 and the system reports corroboration with both quotes attached.

2. A genuine contradiction The annual report gives FY2023 profit after tax as Rs 96.5 crore, consolidated. The investor update gives Rs 112.0 crore for the same year on the same basis. Same subject, same attribute, same stated context, 13.8% apart, and nothing in either document explains the gap.

3. An apparent contradiction that context explains Revenue of Rs 1,240 crore and Rs 870 crore for the same FY2023 looks like a straight conflict until you read the qualifiers: the first is consolidated, the second standalone. Comes out as reconcilable / different_scope. The same documents also produce different_period examples (FY2023 vs FY2022).

The engine handles two more cases of this kind. State change: a director who's active in one document and resigned in a later one gets reconciled by as-of date, not flagged as a conflict. Rounding: two figures that agree closely but not exactly. Both are unit-tested in tests/test_reconcile.py, but neither fires on the sample documents without an API key, because the rule extractor only reads numeric facts and the directorship claims are semantic. Add a key and the LLM extractor picks them up.

4. An extraction failure, and what happens to it

Two real ones, both visible in the running system.

Ungrounded quotes. The extractor doesn't get trusted. Every quote has to be found again in its source page, and one that can't be found is rejected before it's stored and logged to /api/issues with what was dropped and why. That's the mechanism that makes a hallucinated citation fail closed instead of becoming a confident fact. There's a test that feeds in one real quote and one invented one and asserts only the real one survives.

Anaphora the rule extractor can't resolve. The subsidiary filing has the provisional/audited revenue pair that ought to reconcile as different_basis: Rs 1,231 crore provisional against Rs 1,240 crore audited. The rule extractor reads the first one correctly and names the second one audited consolidated figure subsequently, because resolving "the figure" back to "total revenue" needs coreference across sentences. So the two never link, and the layer quietly misses a relation it should have caught.

I left this one visible rather than patching it with a document-specific rule. What happens today: the pair isn't asserted as anything, so nothing wrong gets claimed. Configure a key and the LLM extractor resolves it. The proper fix is a coreference pass plus a second look at any fact whose attribute fails a plausibility check.

There's a third, milder failure worth mentioning. Because the rule extractor takes the attribute from whichever clause is nearest the number, a sentence like "revenue of INR 12.4 billion, up 18% year on year" labels the 18% as revenue instead of growth. It doesn't cause any false relations (a percentage and a money value never get compared) but it's still wrong, and it's the sort of thing the LLM path exists to handle.

What I'd want a reviewer to look at

factlayer/reconcile/rules.py is where the actual thinking lives: the comparability-before-comparison ordering, and the three guards that stop confident wrong answers. factlayer/discriminators.py is tiny but it's the difference between a system that reports a company contradicting itself and one that doesn't.

Repository layout

factlayer/
  models.py           domain model: facts, evidence, relations, qualifiers
  normalize.py        values, currencies, scale words, fiscal periods
  discriminators.py   polarity words fuzzy matching must not smooth over
  ingest.py           PDF -> pages -> passages, and quote re-location
  schema_registry.py  the vocabulary that grows with the corpus
  link.py             candidate generation with blocking
  reconcile/          rules first, LLM for the genuinely ambiguous
  extract/            LLM and rule extractors, table parser, prompts
  llm/                provider-agnostic backends (Gemini/Anthropic/OpenAI/none)
  pipeline.py         orchestration, incremental ingest
  seed.py             the sample layer new sessions start from
  session.py          per-visitor storage, for running this publicly
  api.py              FastAPI service
  cli.py              command line
ui/index.html         single-file interface
tests/                53 tests, no API key required
scripts/              sample PDF generator