Ogen
Ogen is a multi-tenant content operations platform that helps a creator plan,
draft, and publish social content end-to-end:
- Plan campaigns and individual posts with AI assistance grounded in your own content bank.
- Draft in a Markdown-first editor backed by an LLM assistant that can search assets, rewrite posts, and produce platform-specific variants.
- Publish automatically to LinkedIn, X, Facebook, Instagram, Threads, and YouTube via the Zernio broker — durable background jobs handle submission, polling, cancellation, and reconciliation, with a per-post audit log capturing every state change.
This repository is the API — a Go 1.26 monolith (Fiber + Bun + PostgreSQL).
Its background jobs run in
ogen-app/river-service. The React + Vite
SPA lives in its own repository,
ogen-app/ui, and deploys separately.
Contents
Architecture
The architecture below is described with the C4 model at
levels 1 (System Context) and 2 (Containers). The diagrams are rendered
inline as Mermaid so they show up on GitHub; the canonical model is authored as
a Structurizr workspace in
docs/architecture/workspace.dsl (open it in
Structurizr Lite for the full interactive
rendering, including the deployment view).
System context (C4 level 1)
Ogen is a single software system that a creator drives through the web app and
that an operator administers through Harbor. It depends on a handful of hosted
services for language models, publishing, scraping, and email.
C4Context
title System Context — Ogen
Person(creator, "Creator / Operator", "Plans campaigns, drafts posts with AI, schedules publishing")
Person(admin, "Platform Operator", "Manages tenants, secrets, and usage")
System(ogen, "Ogen", "Plans, drafts, and auto-publishes social content grounded in each tenant's content bank")
System_Ext(anthropic, "Anthropic Claude API", "Planning, assistants, quality scoring")
System_Ext(gemini, "Gemini API", "Embeddings, audio transcription, image vision")
System_Ext(zernio, "Zernio", "Social publishing broker")
System_Ext(social, "Social platforms", "LinkedIn · X · Facebook · Instagram · Threads · YouTube")
System_Ext(firecrawl, "Firecrawl", "URL → Markdown scraping")
System_Ext(resend, "Resend", "Transactional & marketing email")
System_Ext(sentry, "Sentry", "Error monitoring + OpenTelemetry traces")
Rel(creator, ogen, "Plans, drafts, schedules", "HTTPS")
Rel(admin, ogen, "Administers", "HTTPS / gRPC")
Rel(ogen, anthropic, "Generates content", "HTTPS")
Rel(ogen, gemini, "Embeds & analyzes assets", "HTTPS")
Rel(ogen, zernio, "Submits & reconciles posts", "HTTPS")
Rel(zernio, social, "Publishes to", "HTTPS")
Rel(ogen, firecrawl, "Scrapes URLs", "HTTPS")
Rel(ogen, resend, "Sends email", "HTTPS")
Rel(ogen, sentry, "Errors & traces", "HTTPS / OTLP")
UpdateElementStyle(creator, $bgColor="#08427b", $fontColor="#ffffff", $borderColor="#052e56")
UpdateElementStyle(admin, $bgColor="#08427b", $fontColor="#ffffff", $borderColor="#052e56")
UpdateElementStyle(ogen, $bgColor="#1168bd", $fontColor="#ffffff", $borderColor="#0b4884")
UpdateElementStyle(anthropic, $bgColor="#999999", $fontColor="#ffffff", $borderColor="#6b6b6b")
UpdateElementStyle(gemini, $bgColor="#999999", $fontColor="#ffffff", $borderColor="#6b6b6b")
UpdateElementStyle(zernio, $bgColor="#999999", $fontColor="#ffffff", $borderColor="#6b6b6b")
UpdateElementStyle(social, $bgColor="#999999", $fontColor="#ffffff", $borderColor="#6b6b6b")
UpdateElementStyle(firecrawl, $bgColor="#999999", $fontColor="#ffffff", $borderColor="#6b6b6b")
UpdateElementStyle(resend, $bgColor="#999999", $fontColor="#ffffff", $borderColor="#6b6b6b")
UpdateElementStyle(sentry, $bgColor="#999999", $fontColor="#ffffff", $borderColor="#6b6b6b")
UpdateRelStyle(creator, ogen, $textColor="#444444", $lineColor="#707070")
UpdateRelStyle(admin, ogen, $textColor="#444444", $lineColor="#707070")
UpdateLayoutConfig($c4ShapeInRow="3", $c4BoundaryInRow="1")
Containers (C4 level 2)
Inside the Ogen boundary, the API monolith is the hub: it serves the SPA over
REST + SSE, owns the control-plane and analytics databases, hands its
background jobs to river-service (and serves their steps), runs the Genkit AI
flows, and exposes an
internal-only gRPC surface that Harbor uses to manage secrets, tenants, plans,
the platform catalog, per-flow model assignments, email, and announcements. PDF, video, document, audio, and image processing are
split out into private-network sidecar services.
C4Container
title Container diagram — Ogen
Person(creator, "Creator / Operator", "Plans, drafts, schedules")
Person(admin, "Platform Operator", "Tenants, secrets, usage")
System_Boundary(ogen, "Ogen") {
Container(spa, "Web App", "React + Vite (ogen-app/ui)", "Planning & Markdown-first drafting UI; separate deploy")
Container(api, "API", "Go 1.26 · Fiber · Bun · Genkit", "REST + SSE, job outbox + job steps, AI flows, operator gRPC surface")
ContainerDb(pg, "Control-plane DB", "PostgreSQL 17 + pgvector", "Domain data, sessions, encrypted secrets, embeddings, job outbox; river-service's River schema")
ContainerDb(ts, "Analytics DB", "TimescaleDB (PG17)", "Usage/cost, activity, follower snapshots — hypertables + continuous aggregates")
ContainerDb(obj, "Object Storage", "S3-compatible (R2 / MinIO)", "Images, PDFs, audio, video, thumbnails, posters")
Container(pdf, "pdf-service", "Go + pdfium", "PDF text, page-aware chunks, thumbnail — private network")
Container(video, "video-service", "Go + ffmpeg", "Video probe + poster frame — private network")
Container(doc, "document-service", "Go (pure)", "Office/text docs → source-anchored chunks — private network")
Container(audio, "audio-service", "Go + ffmpeg + Gemini", "Audio transcode + transcribe → time-anchored transcript — private network")
Container(image, "image-service", "Go + libvips + Gemini", "Image normalize + EXIF-strip + vision extraction (classify/describe/alt) → anchored chunks — private network")
Container(harbor, "Harbor", "Go + Next.js (ogen-app/harbor)", "Ops dashboard: tenants, plans, models, secrets, email, usage")
Container(river, "river-service", "Go · River", "Runs every background job: queue, schedule, outside calls — private network")
Container(riverui, "River UI", "riverui image", "Background-job dashboard")
}
System_Ext(anthropic, "Anthropic Claude API", "LLM")
System_Ext(gemini, "Gemini API", "Embeddings, audio transcription, image vision")
System_Ext(zernio, "Zernio", "Publishing broker")
System_Ext(firecrawl, "Firecrawl", "Scraping")
System_Ext(resend, "Resend", "Email")
System_Ext(sentry, "Sentry", "Errors + traces")
Rel(creator, spa, "Uses", "HTTPS")
Rel(admin, harbor, "Uses", "HTTPS")
Rel(spa, api, "REST + SSE", "HTTPS")
Rel(api, pg, "Reads/writes; migrates at boot", "SQL / pgx")
Rel(api, ts, "Records & reads", "SQL / pgx")
Rel(api, obj, "Stores & serves media", "S3")
Rel(api, pdf, "Parses PDFs", "gRPC")
Rel(api, video, "Probes video", "gRPC")
Rel(api, doc, "Parses documents", "gRPC")
Rel(api, audio, "Transcribes audio", "gRPC")
Rel(api, image, "Ingests images", "gRPC")
Rel(api, anthropic, "Generation / planning / quality", "HTTPS")
Rel(api, gemini, "Embeddings", "HTTPS")
Rel(audio, gemini, "Transcribes audio (multimodal)", "HTTPS")
Rel(image, gemini, "Vision: classify / extract / alt", "HTTPS")
Rel(api, zernio, "Submit / poll / cancel / reconcile", "HTTPS")
Rel(api, firecrawl, "Scrape URL", "HTTPS")
Rel(api, resend, "Send email", "HTTPS")
Rel(api, sentry, "Errors & OTel spans", "HTTPS / OTLP")
Rel(harbor, pg, "Reads", "SQL")
Rel(harbor, ts, "Reads", "SQL")
Rel(harbor, api, "Operator admin", "gRPC")
Rel(api, harbor, "New-tenant webhook", "HTTPS")
Rel(riverui, pg, "Reads job tables", "SQL")
UpdateElementStyle(creator, $bgColor="#08427b", $fontColor="#ffffff", $borderColor="#052e56")
UpdateElementStyle(admin, $bgColor="#08427b", $fontColor="#ffffff", $borderColor="#052e56")
UpdateElementStyle(spa, $bgColor="#438dd5", $fontColor="#ffffff", $borderColor="#2e6295")
UpdateElementStyle(api, $bgColor="#438dd5", $fontColor="#ffffff", $borderColor="#2e6295")
UpdateElementStyle(pg, $bgColor="#438dd5", $fontColor="#ffffff", $borderColor="#2e6295")
UpdateElementStyle(ts, $bgColor="#438dd5", $fontColor="#ffffff", $borderColor="#2e6295")
UpdateElementStyle(obj, $bgColor="#438dd5", $fontColor="#ffffff", $borderColor="#2e6295")
UpdateElementStyle(pdf, $bgColor="#438dd5", $fontColor="#ffffff", $borderColor="#2e6295")
UpdateElementStyle(video, $bgColor="#438dd5", $fontColor="#ffffff", $borderColor="#2e6295")
UpdateElementStyle(doc, $bgColor="#438dd5", $fontColor="#ffffff", $borderColor="#2e6295")
UpdateElementStyle(audio, $bgColor="#438dd5", $fontColor="#ffffff", $borderColor="#2e6295")
UpdateElementStyle(image, $bgColor="#438dd5", $fontColor="#ffffff", $borderColor="#2e6295")
UpdateElementStyle(harbor, $bgColor="#438dd5", $fontColor="#ffffff", $borderColor="#2e6295")
UpdateElementStyle(riverui, $bgColor="#438dd5", $fontColor="#ffffff", $borderColor="#2e6295")
UpdateElementStyle(anthropic, $bgColor="#999999", $fontColor="#ffffff", $borderColor="#6b6b6b")
UpdateElementStyle(gemini, $bgColor="#999999", $fontColor="#ffffff", $borderColor="#6b6b6b")
UpdateElementStyle(zernio, $bgColor="#999999", $fontColor="#ffffff", $borderColor="#6b6b6b")
UpdateElementStyle(firecrawl, $bgColor="#999999", $fontColor="#ffffff", $borderColor="#6b6b6b")
UpdateElementStyle(resend, $bgColor="#999999", $fontColor="#ffffff", $borderColor="#6b6b6b")
UpdateElementStyle(sentry, $bgColor="#999999", $fontColor="#ffffff", $borderColor="#6b6b6b")
UpdateLayoutConfig($c4ShapeInRow="3", $c4BoundaryInRow="1")
The API monolith, up close
The API is a layered Go monolith. A single boot sequence
(cmd/server/main.go) loads config, connects and migrates both databases,
initialises the envelope-encryption cipher + secret store, wires the server, and
starts the HTTP listener alongside the internal gRPC listener.
cmd/server/ # main entrypoint: config → DB + migrations → secrets → server.New → listen
src/
transport/ # inbound surface (CON-291 role grouping)
handlers/ # Fiber HTTP handlers (REST + SSE) with swag annotations
grpc/server/ # internal operator gRPC surface (Secrets, TenantAdmin, PlanAdmin, PlatformAdmin, ModelConfigAdmin, EmailAdmin, AnnouncementAdmin)
grpc/client/{pdf,video,documents,audio,image}/ # gRPC clients we consume (pdf/video/document/audio/image sidecars)
server/ # server.New: wires repos → handlers → workers → routes; Genkit runtime
usecase/ # application orchestration / services — the one home for "where orchestration lives"
post_actions/ campaign_actions/ scheduling/ campaigngoal/ notes/ settings/
brandresolve/ notify/ accountselect/ tenant_actions/ activity/
ideas/ ingest/ connectlink/ loginsecurity/
domain/ # business core
models/ # bun-tagged record structs (single source of truth for the schema)
platforms/ # per-platform attachment + text validation rules, Markdown → social-text flattening
entitlements/ # tier entitlements resolver + nil-safe, fail-open Limiter (caps & gates)
modelconfig/ # flow/slot catalog + tier-aware model resolver
infra/ # persistence + outward adapters
repository/ # narrow per-aggregate persistence layer
database/ # bun.DB factories (control-plane + analytics) and migrations
publishers/ # Publisher abstraction; publishers/zernio is the concrete impl
storage/ secrets/ crypto/ email/ firecrawl/ embedding/ eventhub/ vendors/ geoip/
kernel/ # cross-cutting: config/ logging/ telemetry/ tenantctx/ netguard/ usage/ activity/ background/
jobs/ jobs/queues/ jobs/outbox/ # job outbox + relay, and each job's steps / maintenance pass (run by river-service)
genkit/ # Genkit AI flows (assistants, content plan, quality, embeddings)
analytics/ # post-analytics read models (overview / performers / lessons / per-post)
pgtest/ integration/ # test harness (real Postgres) + //go:build integration suite
Persistence. Domain data, sessions, the encrypted secret table, vector
embeddings (assets_chunks.embedding as a pgvector halfvec(3072)), and the
job outbox all live in the control-plane PostgreSQL 17 + pgvector database.
Bun is the ORM; migrations under src/infra/database/migrations/ run on every
boot. A job is written to job_outbox in the same transaction as the change
that caused it (used for the auto-publish schedule decision), so it exists iff
the change commits. A separate TimescaleDB database
(ANALYTICS_DSN) holds the metering and analytics hypertables + continuous
aggregates, kept isolated so the extension dependency stays quarantined and the
control plane stays vanilla. Analytics is fail-open: a connect/migrate failure at
boot disables it rather than taking down the API.
Background jobs. Jobs run in
river-service, on open-source
River in its own river_service schema. ogen keeps
every decision and write: an enqueue writes job_outbox in the caller's
transaction and the relay delivers it to river-service's JobService. A job
that calls something outside ogen (Zernio, Gemini, Firecrawl, a sidecar) runs as
steps: river-service asks ogen's JobStepService what to do next, makes the
call with a key it leases from ogen, and passes the raw result back, until ogen
says done. Sweeps that only touch ogen's tables run whole through
MaintenanceService. river-service schedules every periodic sweep. The
River UI dashboard reads its schema. The API
refuses to boot without the four job settings below unless JOBS_DISABLED=true.
A database that still has ogen's old in-process River tables gets them retired
at boot: under the migration lock, every unfinished job moves to job_outbox
(keeping its schedule) and River's tables go, in one transaction.
Capabilities
| Area | What's there |
|---|
| Multi-tenancy | Public SaaS signup mints a tenant; all domain rows are tenant-scoped and resolved per request. Users, sessions, invitations, and password reset are first-class. Sign-ins from an unrecognised browser trigger a new-device email (offline GeoIP + relay geofeeds for the location line) whose "This wasn't me" link signs out every session. |
| Campaigns & posts | Campaign types (shared defaults plus tenant-owned types), campaigns, posts, post versions, notes, per-platform validation, quality scoring, and streaming AI assistants. Posts carry an optional content format (how-to, explainer, listicle…) that the assistants respect. |
| Ideas | A workspace Ideas backlog: capture, edit, and triage from an inbox to a verdict (yes / later with a reminder / no). |
| Content bank | Markdown notes, PDF, office/text document, audio (transcribed to time-anchored chunks), image (vision-extracted to source-anchored blocks + alt text), and URL assets (scraped via Firecrawl), with paragraph-/source-aware chunking, Gemini embeddings, and pgvector similarity search used to ground the assistants. Document, audio, and image ingestion run in the document-service / audio-service / image-service sidecars. |
| Attachments | Image, PDF, and video uploads on S3-compatible storage, processed by the image-service / pdf-service / video-service sidecars (EXIF/geo-strip + alt text, thumbnails, page counts, duration/codec/poster). A content-bank image can be attached to a post directly (re-stripped from the original). Post reads expose a cover_url. Per-platform constraints surface as soft validation warnings. |
| Figma plugin | A "Send to Ogen" Figma plugin pushes rendered frames (PNG/JPG) into the content bank as origin: figma image assets, optionally attaching them to a post. It pairs once by polling (the user approves a write key in the web app; the plugin collects a hashed, revocable ogp_ token with its read key), then calls a bearer-only /api/plugins/figma/* API with wildcard CORS. Every call re-checks the member and workspace; members and owners can list and disconnect connections. |
| Zernio auto-publish | Publisher abstraction with Zernio as the concrete impl. Submit / poll / cancel / retry are background jobs; a reconciliation sweeper guards against stuck posts. Headless account connect avoids Zernio's hosted picker. Native threads on X and Threads; YouTube titles go in ; a reschedule whose Zernio post is gone creates a fresh one. Markdown is flattened to platform text at egress and limits count visible length. Deleting a workspace tears down its Zernio profile. |
AI flows (Genkit)
All AI-powered features are Firebase Genkit
flows under src/genkit/flows/. Two long-lived Genkit instances run at boot:
- Embedding instance — the
googlegenai plugin (Gemini Embedding 2). The wrapper is stable for the process lifetime; its backing embedder is rebuilt when gemini_api_key is set or rotated, so the key rotates without restart.
- Anthropic instance — owned by
gkRuntime (src/transport/server/genkit_runtime.go). Hot-rebuildable, so rotating anthropic_api_key re-registers all Anthropic flows without a restart. With no key the runtime stays nil and consuming handlers return 503.
Models are configured per flow and per slot, and are never hardcoded
(src/domain/modelconfig). Each flow declares one or more model slots, and
each slot sets the capabilities a model must have (tools, structured output,
streaming, minimum output/context size, embedding dimensions). Assignments live
in the flow_model_config table. They are resolved at call time, and a
tier-specific row wins over the global default. Operators edit them from Harbor
via ModelConfigAdminService. A write is rejected when the model doesn't meet
the slot's requirements, and it can be checked first with a live probe. Models
are never chosen through env vars. On boot, a slot with no global-default row
yet is seeded from the code defaults in modelconfig.SeedDefaults; after that,
the DB is authoritative. The one exception is embeddings: the embedder runs
EMBED_MODEL (changing it means re-embedding), and the embed slot is seeded
with it for display. In-process chat slots accept only Anthropic models. The
vision and transcribe slots accept only Gemini models, because image-service
and audio-service run the model ogen passes on each request. The assistants'
tool loops (post_assistant planner, campaign_assistant orchestrator) cache
their prompt, whichever model their slot runs.
| Flow | Slot(s) | Seed default |
|---|
content_plan, draft_post, enrich_brief, consistency | main | Claude Sonnet 5.5 |
post_quality | main | Claude Sonnet 5.5 |
post_assistant | planner / writer | Claude Haiku 4.5 / Sonnet 5.5 |
campaign_assistant | orchestrator | Claude Haiku 4.5 |
embed | main (global only, no tier override) | Gemini Embedding 2 (EMBED_MODEL, 3072-dim) |
vision (image-service) | classify / extract / escalate / alt_text | Gemini 3.8 Flash |
transcribe (audio-service) | main | Gemini 3.8 Flash |
An image or audio extraction run resolves its models once, when the run is
created, and keeps them on the extraction row, so retries and every audio
segment use the same model.
| Flow | Purpose | Triggered by |
|---|
embed_asset | Chunks an asset's Markdown (paragraph-aware, with overlap) and writes embeddings into assets_chunks. | Asset save callback. |
process_pdf (job, ingest queue) | Calls pdf-service for text + page-aware chunks + thumbnail, uploads to S3, then chunks + embeds. | PDF upload. |
process_url (job, ingest queue) | Scrapes a URL to Markdown via Firecrawl, mirrors images to S3, stores it as a URL asset, then chunks + embeds. | URL asset creation. |
process_document (job, ingest queue) | Calls document-service for office/text docs (docx/pptx/xlsx/odt/epub/csv/html/eml/…) → source-anchored chunks, then embeds. | Document upload. |
process_audio (job, audio queue) | Probes + tier-gates, normalizes once, segments, transcribes each segment via audio-service (Gemini multimodal), assembles time-anchored transcript chunks, then embeds. Checkpointed + resumable. | Audio upload / finalize. |
process_image (job, image queue) | Presigns the original + a normalized slot, runs the full image-service pipeline (normalize + EXIF-strip + classify + per-shape extraction + description + alt text), persists the extraction/blocks, embeds the searchable text, and snapshots cost. | Image asset upload. |
content_plan | Per-phase, per-platform post drafts for a campaign; parallel K-sized batches against Anthropic with SSE progress. | POST /api/campaigns/:id/generate-draft (SSE). |
post_assistant | Interactive post editor. Haiku planner loop + Sonnet editPost write-tool; streams explanation_delta / content_delta; tool use over the asset library. | POST /api/posts/:id/assistant (SSE). |
campaign_assistant | Campaign-level planning assistant; flows-as-tools (content plan, enrich brief, draft post, consistency check) under a Haiku orchestration loop. enrich_brief edits the campaign's existing brief rather than rewriting it. | Campaign assistant endpoint (SSE). |
Infrastructure & deployment
Ogen runs as a small fleet of containers. In production they are deployed on
Railway; the sidecar services are reached only over
Railway's private network (*.railway.internal), never a public port.
Background jobs run in river-service
(its README has the Railway setup). ogen enqueues them through the job outbox
and serves their steps on the job RPC listener (JOB_RPC_ADDR). Schema
migrations take a Postgres advisory lock, so replicas booting together migrate
one at a time.
Repository family
| Repo | Role |
|---|
ogen-app/ogen | API monolith (this repo). |
ogen-app/ui | React + Vite SPA; deploys independently. |
ogen-app/harbor | Operator dashboard (Go backend + Next.js UI). |
ogen-app/pdf-service | PDF parsing sidecar (CGO + pdfium). |
ogen-app/video-service | Video probing sidecar (ffmpeg/ffprobe). |
ogen-app/document-service | Office/text document parsing sidecar (pure Go). |
ogen-app/audio-service | Audio transcode + transcribe sidecar (ffmpeg + Gemini multimodal). |
ogen-app/image-service | Image normalize + vision-extraction sidecar (libvips/CGO + Gemini vision). |
buf.build/ogen-app/proto | Shared gRPC contracts (tenants, secrets, plans, platforms, modelconfig, email, announcements, pdf, video, documents, audio, image), published as a BSR module and pinned via make proto. |
Runtime topology
- API — stateless Go binary (static,
CGO_ENABLED=0, Alpine runtime). Serves HTTP (ADDR, :3000 in the image) and the internal gRPC surface (GRPC_ADDR, bind :9091 in a container so Harbor can reach it; loopback-only by default). Drops from root to an unprivileged user at entry after fixing KEK-volume permissions.
- Control-plane database — PostgreSQL 17 + the
vector extension (pgvector/pgvector:pg17).
- Analytics database —
timescale/timescaledb:latest-pg17, separate instance (ANALYTICS_DSN).
- Object storage — Cloudflare R2 in production, MinIO locally (any S3-compatible endpoint via
STORAGE_*).
- pdf-service / video-service / document-service / audio-service / image-service — gRPC on
:50051, private network only. audio-service and image-service additionally call the Gemini Developer API — for transcription and vision extraction, respectively (both share gemini_api_key).
- Harbor — reads both databases and calls the API's operator gRPC surface; shares the control-plane Postgres server via its own
harbor database.
- river-service — runs every background job (two Railway services: light and heavy queues), private network only.
- River UI — job dashboard reading river-service's
river_service schema.
Secrets & keys. API keys (Anthropic, Zernio, Gemini, Firecrawl, Resend,
webhook + link-signing secrets) are not required as env vars. They are seeded
from the environment on first boot into the envelope-encrypted secret table and
thereafter read from — and rotated via — the operator gRPC surface, with no
restart. The KEK (envelope-encryption master key) lives at
<OGEN_KEK_PATH>/kek.v1 on a persistent volume (/var/lib/ogen/keys in the
image). Back it up — losing it bricks every encrypted secret. The gRPC
transport is plaintext + bearer-token gated, so it must never cross an untrusted
network.
The Structurizr deployment view renders this
topology onto Railway (open the workspace in Structurizr Lite).
Development
Two workflows, depending on what you're changing:
| You're working on | Use this workflow | What you need |
|---|
| Frontend only | The ogen-app/ui repo — its Vite dev server proxies /api to a locally-running API. | Node + pnpm (see that repo). |
Backend (src/, cmd/) | Backend development — build the API from source; point the UI at http://localhost:9001. | Go 1.26 + Docker. |
The UI targets http://localhost:9001 by default via its Vite /api proxy, so a
locally-running API is picked up automatically.
Backend development
Prerequisites
| Tool | Minimum | Notes |
|---|
| Go | 1.26 | The module sets go 1.26.1. |
| Docker Desktop | 4.x | PostgreSQL, TimescaleDB, MinIO, the sidecars, and the unit/integration suites all run in containers. |
make | any | Build / test / openapi / proto targets. |
buf | any | Only needed to regenerate gRPC stubs (make proto); generated code is committed. |
There is no local PDF/video/audio/image toolchain to install — parsing,
transcription, and vision extraction moved to the pdf-service /
video-service / document-service / audio-service / image-service
sidecars (gRPC). When a sidecar is absent the API degrades gracefully: PDF/video
uploads are accepted but unprobed, while document, audio, and image uploads fail
fast with a clear "not configured" message.
1. Clone and pull deps
git clone https://github.com/ogen-app/ogen.git
cd ogen
go mod download
2. Start the databases
The default DATABASE_DSN points at postgres://ogen:ogen@localhost:5432/ogen.
Bring up Postgres (and the analytics TimescaleDB) via Docker Compose:
docker compose up postgres timescaledb -d
Migrations for both databases run automatically on every boot.
3. KEK (envelope-encryption master key)
The first run generates one under OGEN_KEK_PATH (default ./kek/kek.v1).
Back this file up — losing it bricks every encrypted secret. In deployment,
mount it from a persistent volume.
4. (Optional) seed the Gemini embedding key
Embeddings + semantic search use the hosted Gemini Embedding 2 API. Set
GEMINI_API_KEY to seed it on first boot (thereafter rotate it via the operator
surface). With no key the server still boots: asset saves succeed, but no vectors
are written and semantic search returns nothing.
export GEMINI_API_KEY=AIza... # optional; enables embedding + RAG
5. Run with hot reload
make run # uses `air` for live rebuilds on .go changes
The API listens on http://localhost:9001. Or build + run once:
make build && ./server
6. Configure secrets at runtime
You don't have to set ANTHROPIC_API_KEY / ZERNIO_API_KEY / etc. as env vars —
the server boots without them and surfaces an "integration disabled" state. Seed
them via env on first boot, or manage them through Harbor's Settings → Secrets tab
(which talks to the API's operator gRPC surface). Both are encrypted at rest with
a per-secret DEK wrapped by the KEK, and rotate without restart.
Common Make targets
make build # build the server binary into ./server
make run # air-powered hot reload for the API
make genkit # boot the API under the Genkit dev UI (:4000)
make test # ginkgo unit/handler suites against a throwaway Postgres (race, coverage)
make test-integration # spins up MinIO + Postgres, runs //go:build integration specs
make openapi # regenerate docs/swagger.json from swag annotations
make proto # regenerate gen/ gRPC stubs from buf.build/ogen-app/proto
make tidy # go mod tidy
make docker # build the API Docker image
Configuration
All runtime knobs are env vars, loaded by
envconfig. See
src/kernel/config/config.go for the full, documented list. Highlights:
| Variable | Default | Purpose |
|---|
ADDR | :9001 | HTTP listen address (the Docker image sets :3000). |
DATABASE_DSN | postgres://ogen:ogen@localhost:5432/ogen?sslmode=disable | Control-plane Postgres (pgvector). |
ANALYTICS_DSN | empty | Isolated analytics TimescaleDB. Empty disables usage metering + analytics (fail-open). |
DB_MAX_OPEN_CONNS / DB_MAX_IDLE_CONNS | 25 / 25 | Shared pool sizing for HTTP and job steps; idle connections close after 5 minutes unused. |
GEMINI_API_KEY | empty | Gemini Embedding 2 key — first-boot seed; rotate via the operator surface. Empty disables embedding/RAG. |
EMBED_MODEL / EMBED_DIMENSIONS | gemini-embedding-2 / 3072 | Embedding model + vector width (must match assets_chunks.embedding). |
ANTHROPIC_API_KEY | empty | Claude key (first-boot seed). Models are assigned per flow slot in Harbor, not via env; see AI flows. |
STORAGE_* | empty | S3-compatible storage (R2/MinIO/AWS); empty disables uploads. |
PDF_SERVICE_ADDR / VIDEO_SERVICE_ADDR / DOCUMENTS_SERVICE_ADDR / AUDIO_SERVICE_ADDR / IMAGE_SERVICE_ADDR | empty | gRPC sidecar addresses (*.railway.internal:50051 in prod). Empty disables that pipeline. |
ZERNIO_API_KEY / ZERNIO_BASE_URL | empty / https://zernio.com/api/v1 | Zernio integration; key resolved per call, so rotations land without restart. |
FIRECRAWL_API_KEY | empty | URL-asset scraping; empty disables URL ingestion. |
RESEND_API_KEY / EMAIL_FROM | empty / Ogen <[email protected]> |
Testing
The unit/handler suite runs against a real, throwaway Postgres (no mocks) —
make test boots a disposable pgvector container and every test provisions its
own database through the migration chain. The integration suite
(//go:build integration) brings up MinIO + Postgres via
docker-compose.integration.yml; the embedding + chunking specs additionally hit
the live Gemini API and skip unless GEMINI_API_KEY is set.
make test # unit/handler (throwaway Postgres, race + coverage)
make test-integration # MinIO + Postgres (slow, docker compose)
Observability
| Route | Purpose |
|---|
/api/health | Liveness/readiness. |
/debug/vars | expvar JSON — ogen_jobs_* counters and Zernio API latency (behind session auth). |
/api/posts/:id/log · /api/post-logs | Per-post and cross-post audit history. |
River UI (:9004 in compose) | Running / upcoming / completed / failed jobs per queue. |
Sentry (when SENTRY_DSN is set) | Grouped panics/5xx and end-to-end OpenTelemetry traces (HTTP → Genkit → model → DB → gRPC), continued from the UI's sentry-trace header. See the opentelemetry-tracing skill. |
The full REST reference (OpenAPI) is generated from swag annotations via
make openapi and published from docs/swagger.json on every merge to main.
Running the full stack with Docker Compose
docker-compose.yml runs the API plus its dependencies; optional services are
gated behind profiles so the default
docker compose up stays backend-only.
docker compose up # API + Postgres + TimescaleDB + pdf/document/audio/image sidecars (+ River UI on :9004)
docker compose --profile ui up # + the ogen-app/ui Vite dev server on :9002
docker compose --profile video up # + video-service
docker compose --profile harbor up # + Harbor backend (:9003) & UI (:9005)
The document/audio/image sidecars are wired to the API by default (the api
service defaults DOCUMENTS_/AUDIO_/IMAGE_SERVICE_ADDR to the compose
addresses); set any of them to empty in .env to disable that pipeline. The
audio and image sidecars read GEMINI_API_KEY from .env directly — not from
the API's secret store — so keep it set there, or transcription and vision
calls return Unavailable.
Copy .env.example → .env and fill it in before docker compose up; all
configuration (including seed secrets) is read from there.
Repository skills (.agents/skills/)
The .agents/skills/ directory ships scaffolding skills for
Claude Code (and any agent harness honouring the
SKILL.md convention). Each encodes the project's existing patterns so a new
feature lands looking like the rest of the codebase.
| Skill | When to invoke | What it scaffolds |
|---|
add-entity | Adding a new resource / table / domain entity — a full vertical slice. | Migration up/down, bun-tagged model, narrow Repository + impl, Fiber handler with swag annotations, server.New wiring, ginkgo tests against real Postgres, and an http-client/ example. |
add-genkit-flow | Adding an AI assistant / generator / classifier — anything LLM-driven. | A flow package mirroring post_assistant / content_plan: types, prompt template, run.go with the SSE-streaming closure, optional tools, context cache, runtime init, streaming handler, and tests. |
add-river-job | Adding a background/async task, queue worker, or recurring sweep. | Proto args message, outbox enqueue, the job's steps or maintenance pass, the river-service side, and tests. |
add-grpc-service | Adding a gRPC service/client or a private-network sidecar. | buf-managed proto codegen, (client-)streaming RPCs, health service, graceful shutdown, status-error mapping, and a nil-disabled thin client — the pdf-service pattern. |
email-templates | Adding/editing an email, its copy/design, or the drip schedule. | The Maizzle authoring project + DB seeds + embedded-default → seed-on-boot → DB-wins lifecycle, the [[ ]] Go delimiters, and the brand tokens (CON-154). |
opentelemetry-tracing | Wiring, debugging, or extending tracing / Sentry — missing or unparented spans, broken sentry-trace / traceparent propagation, sampling. | The telemetry.Init wiring point, Sentry-via-OTLP export, the sampler + propagator, and the Fiber c.Context() vs c.UserContext() span-parenting trap. |
All follow the project's conventions — real Postgres tests at the handler layer
(no mocks), authz-before-lookup with 403-over-404, feature/ branch naming — and
refuse to invent new patterns where canonical references already exist. The
SKILL.md files are themselves the spec; read them to audit the patterns the
project considers canonical.
# Inside Claude Code, or any agent harness that loads .agents/skills:
/skill add-entity # then describe the entity
/skill add-genkit-flow # then describe the flow
