3ie Evidence Assistant

---

What it is

The 3ie Evidence Assistant is a multi-tenant retrieval-augmented generation (RAG) platform for researchers and evidence synthesis teams. Users upload study PDFs into their own projects and ask questions across them. Answers cite the source passage, and each citation opens the PDF at the right page.

A separate extraction pipeline reads the results tables in each study and pulls out treatment effects (coefficients, standard errors, p-values, sample sizes), which the API serves to the chat and the Evidence Panel.

---

Status (November 2026)

Area	State
Deployment	Frontend on Vercel, API on Railway, PDF processing on Modal. Last commit: April 2026
RAG answers	Working. Faithfulness 66.5% on a 5-query baseline from January 2026. Target 70%+
Extraction pipeline	12,366 rows from 1,115 results tables in 86 studies. Accuracy is self-measured (no human ground truth yet)
Evaluation	A human-keyed harness from the TVET parser bake-off is being ported. See the TVET Transfer section of the Roadmap
Models	Defaults are 2024-era (gpt-4o-mini, gpt-4o, Cohere rerank-v3.5). Upgrade plan in Roadmap: SOTA Upgrade

Honest disclosure: extraction accuracy has not been validated against human coding. There are no regression tests and no systematic prompt methodology yet. The roadmap's Critical Gaps section lists what is missing and the TVET Transfer section lists how it will be closed.

---

Features

Ingestion

• Docling parsing on Modal (CPU): text, tables (as Markdown) and page numbers. Docling replaced MinerU in November 2026 after a pilot on this corpus (decimal recall 0.988 against 0.502, see evaluation/results/parser_pilot_2026-11-08.md). MinerU on a T4 GPU remains available with --parser mineru
• Eight-step table post-processing: removes empty tables, figures and text fragments misread as tables, merges tables split across pages, grafts orphan headers, and repairs captions from the List of Tables
• Contextual retrieval: each chunk is prefixed with an LLM-written summary of its place in the document
• Parent-child chunking: 256-token chunks for search, 1,024-token parents for context
• Metadata enrichment from Crossref and OpenAlex, and study design classification (22 designs)
• Bibliography import (RIS, BibTeX, CSV) and Zotero sync

Retrieval

• Hybrid search in Supabase Postgres: Cohere embed-v4.0 vectors (1,536 dimensions, pgvector HNSW) plus English full-text search, fused with reciprocal rank fusion in one SQL function (hybrid_search)
• Compact vectors: stored as halfvec (half-precision), about 3 KB per chunk
• Query processing: multi-query RAG-Fusion with reciprocal rank fusion, query decomposition and step-back prompting. HyDE is implemented but disabled, because it lowered faithfulness in testing
• Cohere reranking (rerank-v3.5), MMR diversity filtering and lost-in-the-middle reordering
• Caching: embedding cache and a semantic cache for near-identical questions

Synthesis

• Claim Ledger: a two-pass synthesis that extracts claims and binds each one to a source passage
• Inline citations in [Author, Year, p.X] form, linked to an in-app PDF viewer with highlighting
• Post-hoc verification with MiniCheck natural-language inference
• Per-project settings: model choice, domain instructions, custom prompts with a ten-version history

Extraction pipeline

python -m extraction_pipeline.batch.classify   # classify every table: RESULTS / DESCRIPTIVE / SKIP
python -m extraction_pipeline.batch.extract    # extract values from RESULTS tables
python -m extraction_pipeline.pipeline --study_id 121058356   # one study end to end

Stage	Module	What it does
1. Post-process	extraction_pipeline/postprocessing/mineru.py	Removes MinerU artefacts
2. Caption repair	extraction_pipeline/postprocessing/lot_parser.py	Repairs captions from the List of Tables
3. Classify	extraction_pipeline/classification/	Rules first, then an LLM classifier with few-shot examples
4. Extract	extraction_pipeline/extraction/	Handlers for five table layouts, plus prose and figure fallbacks
5. Store	extraction_pipeline/adapters/supabase_loader.py	Loads the extracted_estimates table in Supabase

Results are served by ten endpoints under /extraction/* (api/routes/extraction.py). Full detail is in extraction_pipeline/README.md.

Admin panel

Overview (usage, costs, projections), users and projects, quality (Langfuse experiments and baselines), documents, processing jobs, and a metadata editor with a PDF viewer.

---

Architecture

graph LR
    subgraph Frontend["Frontend (Vercel)"]
        A[Next.js 16 app] --> U[Chat and workbench]
        A --> V[Evidence Panel]
    end

    subgraph API["API (Railway)"]
        C[FastAPI] --> X["/extraction endpoints"]
    end

    subgraph Processing["Processing (Modal)"]
        E[Docling on CPU]
    end

    subgraph Extraction["Extraction pipeline"]
        Q[Classify tables] --> S[Extract effects]
    end

    subgraph Storage
        I[(Supabase: Postgres, pgvector, auth)]
        K[(Cloudflare R2: PDFs + parser JSON)]
    end

    subgraph Models
        M[OpenRouter: LLMs]
        O[Cohere: embed + rerank]
    end

    A --> C
    C --> M
    C --> O
    C --> E
    E --> K
    E --> I
    K --> Q
    Q --> M
    S --> I
    X --> I
    V --> X

여기까지가 README 앞부분입니다. 나머지는 원문에서 읽을 수 있습니다. README 전문 보기 ↗