Paperless NGX Dedupe

A document deduplication and AI-powered metadata companion for Paperless-NGX. It syncs your documents, identifies duplicates using MinHash/LSH algorithms, and extracts metadata with LLMs through a web UI and REST API.

Features

Deduplication

• Intelligent duplicate detection -- MinHash signatures + Locality-Sensitive Hashing for efficient O(n log n) candidate discovery
• Similarity scoring -- weighted Jaccard similarity and fuzzy text matching with a discriminative penalty that down-scores pairs sharing only boilerplate text
• Side-by-side review -- compare duplicate documents with OCR text diff and resolve them individually or in bulk

AI Metadata Extraction

• Automatic classification -- extract correspondents, document types, and tags from document text using OpenAI models
• Confidence scoring -- per-field confidence scores with supporting evidence snippets so you can review before applying
• Reference-aware -- optionally feeds your existing correspondents, document types, and tags to the LLM to avoid creating duplicates
• Batch processing -- process your entire library or new documents automatically in configurable batches

Platform

• Document sync -- full and incremental sync from your Paperless-NGX instance
• Background processing -- worker threads with real-time progress via Server-Sent Events
• Web UI -- dashboard, document browser, duplicate review, and AI results review
• Batch operations -- bulk review, apply, reject, and delete across duplicates and AI results
• Data export -- CSV duplicate reports and JSON configuration backup/restore
• Observability -- OpenTelemetry traces, metrics, and logs with optional Prometheus scrape endpoint and built-in Paperless-NGX system metrics collector
• Single container deployment -- Docker with embedded SQLite, no external dependencies

Quick Start

1. Create a .env file with your Paperless-NGX connection details:

PAPERLESS_URL=http://your-paperless-instance:8000
PAPERLESS_API_TOKEN=your-api-token-here

You can authenticate with an API token (recommended) or username/password. See Configuration for all options.

2. Start the container:

docker compose up -d

3. Open http://localhost:3000 and go to Settings > Test Connection to verify connectivity.

4. Sync and analyze. From the dashboard, click Sync to pull your documents, then Analyze to detect duplicates. Both run as background jobs with real-time progress.

5. Review duplicates at /duplicates -- compare documents side-by-side and resolve them.

See the Getting Started Guide for a full walkthrough.

Documentation

• Getting Started -- installation, first run walkthrough, upgrading
• Configuration -- environment variables, algorithm tuning, Docker setup
• API Reference -- complete REST API with curl examples
• How It Works -- the deduplication pipeline explained
• Troubleshooting -- common issues and solutions

Development

Discover tasks with just --list, or inspect one with just --show <recipe>.

just setup            # Install dependencies
just run              # Start dev server (http://localhost:5173)
just build            # Production build
just typecheck        # TypeScript type checking
just lint             # Lint
just test             # Run tests

Note: Background jobs (sync, analysis, batch delete) use worker threads that run outside Vite as raw Node.js processes. These do not work with just run because Node.js cannot execute the TypeScript source files directly. Use just docker-dev for local development to test the full workflow.

License

GNU General Public License v3