DataCat
Universal AI-powered data ingestion and form builder platform.
The Pipeline
Every domain — hardware inventory, medical intake, legal discovery, HR onboarding — reduces to three stages:
Ingest Analyze Deliver
───────────────── ───────────────── ─────────────────
Forms, photos, Domain-specific Dashboards for
documents, audio, AI engines with humans. Direct
APIs, custom UIs multi-modal commands for
processing machines & robots
Data in. Intelligence applied. Action out. The rest is implementation detail.
Architecture
Universal Pipeline
The pipeline is domain-agnostic by design. Each stage is independently replaceable.
1. Data Ingestion — Custom forms, multi-modal capture (photos, documents, audio), external APIs, and purpose-built interfaces. The system doesn't care where data comes from; it normalizes everything into a unified schema.
2. AI Analysis — Domain-specific engines with multi-modal processing. GPT-4 Vision handles primary analysis, Google Vision API and AWS Textract serve as fallbacks, Tesseract.js provides offline OCR. Every provider can fail; the system chains them with automatic failover.
3. Action Delivery — Humans get dashboards and reports. Machines get direct commands. The same analysis pipeline feeds both without translation layers.
Form Builder
Drag-and-drop form construction built on @dnd-kit/core. Multi-step forms with per-step validation. Field types: text, textarea, number, date, select, checkbox, radio.
A template library provides starting points. Schema versioning (FormVersions table) ensures backward compatibility — old submissions remain valid when forms evolve. AI-powered schema generation produces Zod validation rules from natural language descriptions.
Multi-Modal Ingestion (Erfassung)
A hybrid service abstraction manages provider failover transparently. Each analyzed field carries a confidence score: title, manufacturer, dimensions, weight, categories, OCR text.
Bull queues handle async processing backed by Redis. WebSocket connections push real-time updates when analysis completes. A 24-hour TTL cache prevents redundant processing of identical photos.
White-Label System
One command rebrands the entire platform:
./scripts/dev/rebrand.sh medical # Healthcare vertical
./scripts/dev/rebrand.sh legal # Legal services
./scripts/dev/rebrand.sh custom "MyApp" "Data Capture"
Environment variables control all branding. Zero code changes required. Presets ship for: default, HR, medical, legal, government, and generic verticals.
5-Layer Data Integrity
Each layer catches what the previous one missed:
- Client-side Zod validation — Immediate UX feedback
- Server route validation — Schema enforcement at the boundary
- Prisma constraints — Database-level guarantees
- Property-based tests — Playwright and Vitest verify invariants
- Post-ingest checks — Background jobs catch drift
API Architecture
tRPC provides end-to-end type safety from frontend to backend — no generated clients, no schema drift.
Current structure: monorepo with Next.js 16 frontend (port 3000) and Express 5.1 backend (port 5001). Consolidating toward a unified Next.js App Router + tRPC architecture.
Bull job queues manage async processing. Socket.io handles real-time updates. Redis backs both.
Tech Stack
| Layer | Technology |
|---|
| Frontend | Next.js 16, React 19, TypeScript 5, Tailwind 4 |
| State | Zustand, @dnd-kit (drag-and-drop) |
| Backend | Express.js 5.1, tRPC 11.4 |
| Database | PostgreSQL 14+ (Prisma 6), Redis |
| AI | OpenAI GPT-4 Vision, Google Vision API, Tesseract.js |
| Jobs | Bull 4.16 (Redis-backed queues) |
| Real-time | Socket.io 4.8 |
| Testing | Vitest (unit), Playwright (E2E) |
| Deployment | Docker, GitHub Actions (self-hosted on the Hetzner box behind Caddy) |
Prerequisites
- Node.js 20+
- PostgreSQL 14+
- Redis 7+
- Docker (optional, for containerized setup)
Setup
git clone https://github.com/bitbaum/datacat.git
cd datacat
cp backend/.env.example backend/.env # Configure database, Redis, API keys
cp frontend/env.example frontend/.env.local
pnpm install # Also run in frontend/ and backend/
pnpm -C backend run migrate # Run Prisma migrations
pnpm run dev # Starts frontend (3000) + backend (5001)
Environment Variables
Configure at minimum:
DATABASE_URL=postgresql://user:pass@localhost:5432/datacat
REDIS_URL=redis://localhost:6379
OPENAI_API_KEY=sk-...
Docker
docker compose up -d # PostgreSQL, Redis, app
Project Structure
datacat/
frontend/ # Next.js 16 app (App Router, Zustand, Tailwind)
backend/ # Express 5.1 + tRPC backend (flat layout)
prisma/
schema.prisma # Database schema (SSOT for types)
migrations/ # Version-controlled migrations
scripts/
dev/
rebrand.sh # White-label rebranding
tests/ # Playwright E2E suites
Design Principles
Single source of truth. Prisma schema defines the data model. Zod schemas derive from it. Types flow from schemas. Nothing is defined twice.
Fail gracefully, fail loudly. Every AI provider will go down. The failover chain handles it without user intervention. When all providers fail, the system tells you exactly what happened — no silent data loss.
Configuration over code. Adding a new white-label vertical is a config file, not a fork. Adding a new form field type is a registry entry, not a component rewrite.
Validate at every boundary. User input is hostile. API input is hostile. Even internal service communication validates. Five layers exist because no single layer is sufficient.
License
MIT
