GIGO — Garbage In, Gold Out
The AI-powered universal data adapter. Send any JSON to a webhook, get back a payload that matches your schema — renamed, retyped, restructured — and optionally forwarded straight to its destination.
No field-by-field mapping rules. You show GIGO one example of the output you want; the AI understands the meaning of incoming data and reshapes it accordingly.
{ "user_first_name": "jean", { "first_name": "Jean",
"user_MAIL": "[email protected]", ──► "email": "[email protected]",
"amount": "49,90 €", "amount_cents": 4990,
"infos": { "ville": "Lyon" } } "city": "Lyon" }
Why
Every service speaks its own dialect: firstName here, first_name there, amounts as strings, nested structures nobody agreed on. Connecting N sources to M destinations means writing and maintaining N×M brittle integrations. GIGO replaces them with a single hub: one webhook per target schema, AI does the translation, full audit trail included.
Features
- Adapters by example — paste a sample of the JSON you want, or let AI extract the schema from API docs or a documentation URL
- Two AI providers — OpenAI (GPT-4o structured outputs) or Anthropic (Claude, strict tool use + prompt caching), selectable per adapter
- Live playground — test payloads in the UI: see the transformed output, latency, and token usage before wiring anything
- Destination forwarding — auto-deliver transformed data (POST/PUT/PATCH) with Bearer / API-Key / Basic auth, credentials encrypted at rest (AES-256-GCM)
- Production-grade webhook — per-adapter secret (timing-safe verification), rate limiting, 1MB payload cap, timeouts and retries everywhere
- Full observability — every transformation logged with input/output, durations, tokens, trace IDs; cursor-paginated log explorer with live refresh and one-click replay
- Bilingual UI — English / French, switchable at runtime
- Multi-tenant — Supabase Auth, all data scoped per user
Quickstart
Prerequisites
- Bun 1.x
- A Supabase project (auth + Postgres)
- An OpenAI and/or Anthropic API key — bring your own key: each user stores their own keys in Settings (encrypted at rest). Optionally, the operator can set
OPENAI_API_KEY / ANTHROPIC_API_KEY env vars as a server-wide fallback
Setup
git clone <repo-url>
cd universal-data-adapter
bun install
cp .env.example .env # then fill in your values
bun run db:deploy # apply database migrations
bun run db:generate
bun run dev # → http://localhost:3000
ENCRYPTION_KEY (min 16 chars) is required — the app refuses to start without it. It encrypts destination credentials with AES-256-GCM.
Docker
docker compose up -d # see docker-compose.yml for configuration
Usage
- Sign up, then create an adapter: paste an example of your target JSON (or generate it from docs).
- Pick the AI model, optionally configure a destination + webhook secret.
- Test it in the Playground on the adapter page.
- Point any service at your webhook:
curl -X POST https://your-instance/api/webhook/<ADAPTER_ID> \
-H "Content-Type: application/json" \
-H "X-Webhook-Secret: whsec_..." \
-d '{"any": "json", "shape": "works"}'
Response:
{
"status": "success",
"trace_id": "…",
"data": { "…": "transformed payload" },
"meta": {
"duration_ms": 1250, "transform_duration_ms": 1180,
"provider": "anthropic", "model": "claude-sonnet-5",
"input_tokens": 412, "output_tokens": 96
}
}
Errors are typed: 401 INVALID_SECRET, 413 PAYLOAD_TOO_LARGE, 429 RATE_LIMITED (+ Retry-After), 422 MAX_TOKENS, 504 TIMEOUT — always with a trace_id you can look up in the logs.
Architecture
Sender ──POST──► /api/webhook/[id]
│ secret check (timing-safe) → rate limit → size cap
▼
shared pipeline (src/lib/pipeline.ts)
│ transform → validate → forward → log → usage
▼
provider layer (src/lib/providers/)
├─ openai.ts GPT-4o, strict structured outputs
└─ anthropic.ts Claude, forced strict tool use + prompt caching
The same pipeline powers the public webhook, the authenticated playground (POST /api/adapters/[id]/test) and log replay — three entry points, one behavior.
| Layer | Tech |
|---|
| Framework | Next.js 14 (App Router), TypeScript strict |
| Database | PostgreSQL + Prisma (versioned migrations) |
| Auth | Supabase Auth |
| AI | openai + @anthropic-ai/sdk behind a common TransformProvider interface |
| i18n | next-intl (cookie mode, no URL prefix) |
| Tests | Vitest — bun run test |
| Tooling | Bun |
Security notes
- Destination credentials: AES-256-GCM, versioned ciphertext format, no fallback key.
- Webhook secrets: compared timing-safe (SHA-256 +
timingSafeEqual).
- User-supplied URLs (doc import, destinations) go through an SSRF guard: DNS resolution, private/link-local/metadata IP blocking, re-validation on every redirect, streamed 2MB response cap.
- Rate limiting: fixed window in Postgres — serverless-safe, zero extra infrastructure.
Development
bun run test # 61 unit/integration tests
bun run test:watch
bunx tsc --noEmit # typecheck
bun run build # production build
bun run db:studio # browse the database
See CONTRIBUTING.md for guidelines.
Roadmap
License
MIT
