Moul
moul is a lightweight, self-contained dynamic database, authentication, and background job processing engine in Go, inspired by PocketBase and Elixir's Oban.
Table of Contents
Key Features
- Dynamic Moul (Tables): Create, list, update, and delete database tables and schemas at runtime via HTTP API or TUI.
- Dynamic Record CRUD: Perform complete CRUD operations on any dynamic moul using raw JSON payloads governed by HCL-like authorization rules.
- Multi-Factor & Modern Auth: Built-in support for Password, Email OTP, WebAuthn Passkeys, Social OAuth2 (GitHub, Google, Apple), and OAuth2 Device Flow.
- Rule Authorization Engine: Enforce robust access rules (e.g.
@request.auth.id != "" or @collection.userRoles.userId = @request.auth.id) dynamically, featuring datetime macros, field modifiers, wildcard matching, and database helper functions.
- Background Worker Engine: High-performance, SQLite-backed asynchronous background job processor (inspired by Elixir's Oban) with queue priorities, automatic retries with exponential backoffs, and immediate dispatch triggers.
- Single Binary SQLite: Driven by
github.com/pocketbase/dbx and the CGO-free modernc.org/sqlite driver for lightweight, zero-configuration local development and deployment. On first startup, all reserved system tables prefixed with _* (_moul, _visits, _requests, _settings, _rootUsers, _feature_flags, _certmagic, _revoked_tokens, _workers) are automatically created.
- First-Party Analytics & Session Tracking: Create
analytic moul that automatically track events and sessions. Parses client headers (IP, User-Agent, Referrer, UTM parameters) to resolve browser, OS, device, referring domain, and marketing campaign parameters, including optional MaxMind GeoIP2 resolution.
- Default Request Tracking: All HTTP requests are automatically tracked via a global middleware. Visitor sessions are deduplicated in
_visits, and per-request data (method, path, status code, response time) is batch-inserted asynchronously into _requests for zero-latency-impact observability. The request log can be turned off, sampled, or scoped with request log settings.
- TUI Admin Console: Full-featured Terminal User Interface (TUI) built with Charm's Bubble Tea to manage schemas, records, worker queues, analytics, email templates, and system settings without requiring a browser.
Technical Stack
Local Development Guide
Prerequisites
- Go: Version 1.26 or higher
- Air (Optional, for live reload): Install via
go install github.com/air-verse/air@latest or brew install air
- MinIO & MinIO Client (
mc) (Optional, for local S3 testing): brew install minio minio-client on macOS
Environment Configuration
The engine reads environment variables from a .env file in the project root if present. Default environment setup for local development:
MOUL_ENV=development
MOUL_PORT=8090
MOUL_JWT_SECRET=test-secret-key-for-unit-tests-1234
MOUL_ADMIN_KEY=test-admin-key-1234
MOUL_ENCRYPTION_KEY=test-encryption-key-for-unit-tests-1234
MOUL_DB_PATH=moul-local.db
Run Local Server
Start the Echo HTTP server and background worker engine on port :8090 (this automatically creates or opens the local SQLite database file moul-local.db):
make run
# Or directly via Go:
go run cmd/moul/main.go start
Live Reloading (Hot Reload)
For active local development with instant code recompilation on change:
make dev
# Runs air using .air.toml
Unified Multi-Process Dev Runner
Start both the Go backend daemon (with live reload) and the Web Admin UI (Vite dev server) concurrently:
make dev-all
Database Seeding & Mock Fixtures
Populate your local SQLite database with realistic demonstration collections (users, posts, categories, tasks_queue, events), demo records, relations, worker jobs, and OpenFeature flags:
make seed
# Or directly via CLI:
go run cmd/moul/main.go seed --db moul-local.db
TypeScript Type Generation (typegen)
Generate end-to-end TypeScript interfaces directly from dynamic collection schemas:
make typegen
# Outputs to ui/src/types/schema.d.ts
Rule Expression Evaluation Sandbox
Test, validate, and benchmark rule expressions against mock JSON payloads:
go run cmd/moul/main.go test-rule \
--rule="authorId = @request.auth.id && viewsCount > 100" \
--record='{"authorId": "usr_admin_001", "viewsCount": 1420}' \
--auth='{"id": "usr_admin_001"}'
Background Worker DLQ Management
Inspect discarded jobs and retry failed worker executions:
# List discarded jobs in dead-letter queue (DLQ)
go run cmd/moul/main.go worker list-failed tasks_queue
# Retry all failed jobs (or a specific job by ID)
go run cmd/moul/main.go worker retry tasks_queue [optional_job_id]
Data Import & Export (CSV & JSON)
Export collection records or import new data directly via the CLI with support for CSV (RFC 4180) and JSON (with optional schema envelopes):
# Export collection records to CSV or JSON (local SQLite)
moul export posts --format=csv --out=posts.csv
moul export posts --format=json --out=posts.json --schema
# Export from a remote moul server via Admin Key
moul export posts --server=http://localhost:8090 --admin-key=test-admin-key-1234 --format=json --out=posts.json
# Import records into a collection with conflict resolution
moul import posts posts.csv --mode=upsert
moul import posts posts.json --mode=replace --error-strategy=continue
# Import into a remote moul server
moul import posts posts.csv --server=http://localhost:8090 --admin-key=test-admin-key-1234 --mode=upsert
Local S3 Storage (MinIO)
For testing file uploads and Litestream database backups locally against an S3 API:
-
Start MinIO server:
make minio-start
Starts MinIO server storing data in tmp/minio (Console available at http://localhost:9001, credentials: minioadmin / minioadmin).
-
Configure MinIO Client (mc):
In a separate terminal, register the moul-local alias:
make minio-setup
-
Create Storage Buckets:
mc mb moul-local/moul-bucket
mc mb moul-local/moul-litestream
Testing & Verification Flows
moul includes self-contained in-process integration tests, unit test suites, and automated cURL verification flows.
1. Self-Contained E2E Flow (No External Server Needed)
# Run full CRUD, auth, worker, analytics, and rules verification in-process
make test-e2e
2. Unit & Integration Tests
# Run all Go package tests
make test-go
# Run tests with code coverage output
make test-coverage
# Run TUI tests
make test-tui
3. Automated cURL Flow Scripts (Live Server)
Ensure the server is running (make run) in a separate terminal before executing live curl flow tests:
- Verify Dynamic CRUD, Auth, and Rules Enforcement:
make test-flow
- Verify Asynchronous Background Worker Queue:
make test-worker
- Verify First-Party Analytics and Visit Resolution:
make test-analytics
4. SSE Scalability & Backpressure Benchmarking
The dedicated cmd/sse-bench tool stress-tests Server-Sent Events across 100, 1,000, and 10,000 connections with automated leak detection, slow-consumer backpressure verification, and interactive HTML dashboards:
# Run in-process engine benchmark suite (100, 1k, 10k connections)
make bench-sse
# Run network benchmark over HTTP/TCP sockets
make bench-sse-network
# Run CLI with custom baseline regression checks and HTML export
go run cmd/sse-bench/main.go --mode=engine --targets=100,1000,10000 --export-html=report.html --compare-with=baseline.json
Documentation & API Specification (/docs)
moul maintains an accurate OpenAPI 3.0 specification serving interactive API documentation directly from the running engine.
Interactive Runtime /docs Endpoint
When the server is running (make run), interactive API documentation is accessible in your browser:
OpenAPI Specification Files
The OpenAPI spec is available in both YAML and JSON formats for exporting to Postman, Insomnia, or API SDK generator tools:
Embedded OpenAPI Spec in Go
The specification file is embedded into the compiled binary via Go's //go:embed openapi.yml directive in docs/docs.go. This ensures single-binary deployments serve live API docs without requiring external static asset files.
Extensibility & Custom Workers
Web Admin Console (/_moul_/)
Overview
moul features a high-performance, single-binary Web Admin Console mounted at /_moul_/ (with automatic redirect from /_moul_). It provides a rich, responsive browser-based dashboard to manage collection schemas, dynamic records, real-time SSE subscriptions, analytics, background worker queues, feature flags, and server settings.
Frontend Architecture & Stack
- UI Component System: Moul UI (
@moul-dev/ui) built on React Aria Components and StyleX for accessible, zero-runtime atomic UI components.
- Background Worker Engine & Jobs Queue Monitor: Real-time worker jobs queue table with status filters (
available, executing, completed, retryable, discarded), execution attempt metrics, job payload inspector drawer, and instant Retry and Discard actions.
- Visual Telemetry & Analytics Dashboard: Built-in interactive AreaChart (requests over time categorized by success vs error) and TopList (top endpoints & referrers) powered by
@moul-dev/ui alongside live server latency, throughput, and error rate stats.
- Real-Time Stream Viewer: Server-Sent Events (SSE) listener integrated with
@moul-dev/ui's high-performance Logs stream viewer with syntax highlighting, search/level filters, drawer payload inspector, and auto-scroll follow.
- Data Import & Export Modals: Accessible
@moul-dev/ui modals for records with drag-and-drop file upload, CSV & JSON format auto-detection, schema envelope controls, conflict mode selectors (upsert, insert, replace), and atomic/continue error handling with detailed row error breakdown.
- Feature Flag Evaluation Playground: Live evaluation sandbox to test flag targeting gates, percentage rollouts, and actor/group targeting rules against customizable JSON contexts with instant reason code breakdowns.
- Settings Unsaved State Protection: Form divergence tracking (
isDirty) with visual warning badges and AlertDialog confirmation guards protecting against accidental loss when configuring S3, Litestream, rate limits, OAuth, or email.
- Actionable System Overview & Live Collection Metrics: Engine overview with live record count badges per schema and quick "+ New Record" actions.
- Enhanced Collection Creation & Schema Designer: Dual-tab creation drawer ("General & Fields" and "API Access Rules") with type-based schema templates, live field modeling across 11 data types, smart rule token autocomplete (
@request.*, @collection.*, schema fields), quick preset chips, and an interactive Rule Documentation modal dialog.
- : Complete visual relationship builder directly in Schema Designer supporting , , , self-referencing tree structures, on-delete referential integrity policies (, , ), and interactive linked record pickers with inline relation expansion.
Local UI Development & Live Reloading
For frontend development with instant Hot Module Replacement (HMR) and API proxying:
# 1. Install frontend dependencies
make ui-install
# 2. Start the moul backend engine in one terminal
make run
# 3. Start the Vite dev server in another terminal (proxies /api to localhost:8090)
make ui-dev
Open http://localhost:5173/moul/ in your browser.
Single-Binary Embedding in Production
In production, the compiled SPA bundle is embedded directly into the Go binary at compile time via Go's //go:embed all:dist directive in pkg/ui/ui.go.
# Build standalone moul binary with embedded docs and Web Admin Console
make build
# Start single-binary server
./bin/moul start
Access the Web Admin Console at http://localhost:8090/moul/. All HTML5 SPA sub-routes (e.g. /_moul_/collections/posts, /_moul_/settings) are resolved seamlessly via Echo's embedded SPA fallback router.
Moul TUI Admin Console (moul-ctl)
Overview
moul-ctl is a modern, self-contained Terminal User Interface (TUI) built with Charm's Bubble Tea ecosystem (bubbletea, bubbles, lipgloss). It provides full administration over your moul engine directly from the command line without opening a browser.
Build and Run
Run via Go
make ctl
# Or run via convenience subcommand in moul:
./bin/moul ctl
# Or directly from source:
go run cmd/moul-ctl/main.go
Build Compiled TUI Binary
make build-ctl
./bin/moul-ctl
Connection Setup & Config Storage
On initial startup, moul-ctl prompts for connection credentials:
- Server URL: Server address (defaults to
http://localhost:8090).
- Admin Key: The security key configured on the server (
MOUL_ADMIN_KEY).
Credentials and connection state are securely saved to ~/.config/moul.json (with fallback to system keyring where available) for automatic re-connection on subsequent launches.
TUI Features & Capabilities
- Dashboard: High-level system overview displaying collection counters, active workers, visit statistics, and quick system links.
- Real-Time Inline Form Validation: Built-in validation for email format, password requirements (minimum 8 characters with confirmation matching during root setup and Auth collection record creation), URLs, numeric inputs, and JSON syntax.
- Collection Schema Management: Create new collections (
base, auth, worker, analytic), add/remove fields, choose field types, and write HCL access control rules.
- Dynamic Record CRUD Console: Browse collection records with pagination (
> / <), live search and server-side filtering (/), inspect JSON payloads, copy payload directly to clipboard (c), add new records, edit existing records, and delete records.
- Background Workers Monitor: Real-time 3-second auto-refreshing monitor of background job queues (
executing, available, completed, discarded). Inspect job parameters, copy details (c), force-retry failed jobs (r), or discard jobs (c).
- Analytics & Visits Observatory: Authenticate user credentials (
l) to inspect live site visit logs (_visits) with date range filtering (t: All Time, 24h, 7d, 30d), OS, browser, device type, resolved GeoIP location, landing page, and UTM campaign tracking parameters.
- Email Templates Editor: Customize transactional email templates (OTP verification, Password Reset) per auth collection and send live test emails.
- System Settings Panel: Comprehensive configuration over dynamic system areas with instant server-side hot-reloading (
POST /api/admin/reload or R in TUI): S3 Object Storage, Litestream Continuous Backups, Sliding Window Rate Limiting Rules, Root User IP Whitelist, Transactional Email Delivery (SES, Resend, Mailgun, SendGrid, Cloudflare, Console), OAuth2 Identity Providers (GitHub, Google, Apple), and Root Administrator Credentials.
Keyboard Controls Guide
| Screen | Key | Action |
|---|
| Global | ctrl+c | Exit program |
| Connection Screen | Tab / Shift+Tab | Navigate inputs |
| Enter | Connect to server |
| Dashboard | ↑/↓ or k/j | Navigate collections and options |
| Enter / l / → | Open selected item |
| r | Refresh schema list from server |
| Esc | Disconnect and return to login screen |
| Record List | ↑/↓ or k/j | Scroll record list |
| > / < (or . / ,) | Next / Previous page |
| / | Activate live search & filter query |
| Enter / v | Inspect detailed JSON payload |
| x | Export collection records (CSV / JSON) |
| i | Import collection records (interactive form) |
| n | Create new record |
|
Programmatic Go API
In addition to HTTP endpoints, backend processing and custom background worker tasks can be implemented programmatically in Go.
1. Register and Start Worker Engine
import (
"context"
"github.com/moul-dev/moul-dev/pkg/worker"
)
// 1. Initialize worker engine
workerEngine := worker.NewEngine(dbConn)
// 2. Register custom execution handler
workerEngine.Register("SendEmail", func(ctx context.Context, job *worker.Job) error {
to := job.Args["to"].(string)
subject := job.Args["subject"].(string)
// Execute custom background logic...
println("Sending email to " + to)
return nil
})
// 3. Start processing loop with context cancellation
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
workerEngine.Start(ctx)
defer workerEngine.Stop() // Gracefully awaits running tasks to complete
2. Enqueue Job Programmatically from Go
jobOpts := map[string]interface{}{
"worker": "SendEmail",
"priority": 1,
"args": map[string]interface{}{
"to": "[email protected]",
"subject": "System Alert",
},
}
job, err := workerEngine.Enqueue(context.Background(), "background_tasks", jobOpts)
3. Embedding pkg/app with Custom HTTP Routes, Workers, and Web Admin Console
pkg/app allows embedding the complete moul server into custom Go binaries with tailored HTTP endpoints, background workers, and the pre-built Web Admin Console:
package main
import (
"context"
"net/http"
"github.com/labstack/echo/v5"
"github.com/moul-dev/moul-dev/pkg/app"
"github.com/moul-dev/moul-dev/pkg/worker"
)
func main() {
// The Web Admin Console is automatically bundled and served at "/_moul_/"
moulApp := app.New(app.Config{
Version: "1.0.0-custom",
})
// Optional: customize API prefix, admin mount prefix, custom SPA filesystem, or disable
// moulApp.WithAPIPrefix("/v1") // customize API prefix (use "" for root)
// moulApp.WithAdminPrefix("/dashboard")
// moulApp.WithAdminUI(myCustomFS)
// moulApp.DisableAdminUI()
// Register custom HTTP route
moulApp.RegisterRoute("GET", "/api/custom/ping", func(c *echo.Context) error {
return c.JSON(http.StatusOK, map[string]string{"status": "pong"})
})
// Access raw Echo router via hook
moulApp.OnRouterInit(func(router *echo.Echo) error {
router.GET("/api/custom/health", func(c *echo.Context) error {
return c.String(http.StatusOK, "OK")
})
return nil
})
// Register custom background worker
moulApp.RegisterWorker("CustomTask", func(ctx context.Context, job *worker.Job) error {
return nil
})
moulApp.Start(context.Background())
}
Detailed guides & runnable examples:
4. Programmatic Analytics API
import (
"context"
"github.com/moul-dev/moul-dev/internal/analytics"
)
params := &analytics.EventParams{
VisitToken: "visit-123",
VisitorToken: "visitor-456",
Name: "payment_completed",
Properties: map[string]interface{}{
"amount": 99.9,
},
}
event, err := analyticsEngine.Track(context.Background(), "events", params)
5. Write Transactions (app.WriteTx)
Every transaction Moul opens starts with BEGIN IMMEDIATE, so it owns SQLite's write lock before its first read. Use app.WriteTx for any code that reads and then writes: it cannot fail with SQLITE_BUSY because another connection committed between the read and the write, and it re-runs the transaction (with backoff) if the lock stays busy past the 5 s busy timeout. fn may run more than once, so keep it free of side effects outside tx, and keep it short.
err := moulApp.WriteTx(func(tx *dbx.Tx) error {
var stock int
if err := tx.NewQuery("SELECT stock FROM products WHERE id = {:id}").Bind(dbx.Params{"id": id}).Row(&stock); err != nil {
return err
}
_, err := tx.Update("products", dbx.Params{"stock": stock - 1}, dbx.HashExp{"id": id}).Execute()
return err
})
6. Record Hooks & the Records API
Record hooks run for every record write, whether it arrives over HTTP or from Go code, so application logic lives in one place. Register them before Bootstrap:
a.OnRecordBeforeCreate(func(e *app.RecordEvent) error {
// e.Record holds the values about to be stored; change them or return an error to refuse.
// Return an *echo.HTTPError to choose the status (400 otherwise).
return nil
}, "posts")
a.OnRecordAfterCreate(func(e *app.RecordEvent) error {
// e.Record is the stored record; e.Auth is the caller (nil for admins and Go code).
return notifyFollowers(e.Context, e.Record)
}, "posts", "comments")
OnRecordBefore{Create,Update,Delete} and OnRecordAfter{Create,Update,Delete} take the collections to watch (none = all). Before hooks of a delete also run for cascaded deletes.
a.Records() writes through the same pipeline as the HTTP API (validation, rules, webhooks, realtime events, file tracking and hooks), with admin rights; AsUser(collection, id) applies the collection rules as that user:
post, err := a.Records().Create(ctx, "posts", map[string]interface{}{"title": "Hi", "author": userID})
alice, err := a.Records().AsUser("users", "alice")
page, err := alice.List(ctx, "posts", url.Values{"filter": {"status = 'draft'"}})
Writes your code makes with raw SQL or in its own transaction can be announced after commit, so hooks, webhooks and realtime subscribers see them like API writes:
err := a.Records().Notify(ctx, app.RecordCreate, "notifications", id, nil)
err = a.Records().Notify(ctx, app.RecordDelete, "notifications", id, deletedRecord)
7. Token Lifetime
Auth tokens live 24 hours by default. Configure them with Config.TokenTTL / app.WithTokenTTL (or MOUL_TOKEN_TTL), per collection with app.WithCollectionTokenTTL("users", 30*24*time.Hour) (root users: MOUL_ROOT_TOKEN_TTL), and the refresh grace with app.WithTokenRefreshGrace (or MOUL_TOKEN_REFRESH_GRACE, default 7 days): an expired token can be exchanged at POST /api/moul/:name/refresh only within the grace period, and only once.
Built-in Model Context Protocol (MCP) Server
moul includes a built-in MCP server powered by github.com/mark3labs/mcp-go. This allows AI assistants (Claude Desktop, Cursor, AI agents) to inspect, query, and manage your dynamic database, background jobs, feature flags, and host metrics.
Transport Modes
- Stdio Transport Mode (
moul mcp or <custom-binary> mcp): Runs directly as a CLI subcommand over standard input/output. Seamlessly supported in standalone moul and any custom binary extending pkg/app. Requires zero secrets or authentication tokens as it executes with the privileges of the local user.
- Streamable HTTP & SSE Mode (
/api/mcp): Enabled automatically on moul start (or your embedded binary). Supports direct JSON-RPC POST requests (MCP 2025 Spec) and SSE streaming.
HTTP Authentication Methods
When accessing /api/mcp over HTTP/SSE, two primary header authentication methods are supported:
- Bearer Token Header (Recommended):
Authorization: Bearer <MOUL_ADMIN_KEY> (standard for Cursor, Windsurf, Claude Code).
- Admin Key Header:
X-Admin-Key: <MOUL_ADMIN_KEY> (ideal for API gateways, proxies, and cURL scripts).
Keys in the query string (?adminKey=, ?admin_key=, ?api_key=, ?token=) are rejected with 400 Bad Request so they never land in proxy logs or browser history.
Integration Examples
Stdio Mode (Claude Desktop, Cursor, Antigravity)
Claude Desktop Configuration (claude_desktop_config.json):
{
"mcpServers": {
"moul": {
"command": "/path/to/moul",
"args": ["mcp"],
"env": {
"MOUL_DB_PATH": "/path/to/moul-local.db"
}
}
}
}
Cursor IDE (.cursor/mcp.json):
{
"mcpServers": {
"moul": {
"command": "moul",
"args": ["mcp"]
}
}
}
Streamable HTTP / SSE Mode (http://localhost:8090/api/mcp)
Bearer Token Authentication (Recommended):
{
"mcpServers": {
"moul": {
"type": "http",
"headers": {
"Authorization": "Bearer ADMIN_SECRET_KEY"
},
"url": "http://localhost:8090/api/mcp"
}
}
}
Admin Key Header Authentication (X-Admin-Key):
{
"mcpServers": {
"moul": {
"type": "http",
"headers": {
"X-Admin-Key": "test-admin-key-1234"
},
"url": "http://localhost:8090/api/mcp"
}
}
}
Custom Apps (Google Gemini)
To connect moul as a Custom Connected App in Google Gemini Apps (gemini.google.com/apps):
- In Gemini, navigate to Settings > Connected Apps > Custom apps.
- Click Add a custom app and enter your
moul MCP URL (e.g., https://your-moul-instance.com/api/mcp).
- Click Next.
moul automatically serves standard RFC 9728 Protected Resource Metadata and RFC 8414 Authorization Server Metadata via /.well-known/.- Gemini automatically discovers endpoints and registers via RFC 7591 Dynamic Client Registration.
- In the authorization consent screen, enter your root admin credentials (username/email and password) or your
MOUL_ADMIN_KEY, then click Authorize Google Gemini.
- Once authorized, Gemini connects seamlessly over Streamable HTTP with an OAuth 2.1 Bearer access token.
(Optional fallback: If you configure manual credentials, click Show more under Advanced features in Gemini, enter your Client ID and Client Secret, and complete the authorization consent screen).
Available MCP Tools
| MCP Tool Name | Description |
|---|
moul_list_collections | List all dynamic collection schemas and tables |
moul_get_collection | Get detailed field schema and rules for a collection |
moul_create_collection | Create a new dynamic collection schema and SQLite table |
moul_delete_collection | Delete a collection and drop its physical table |
moul_list_records | Query paginated records from any collection |
moul_get_record | Retrieve a single record by ID |
moul_create_record | Insert a new dynamic record |
moul_update_record | Update an existing record |
moul_delete_record | Delete a record by ID |
moul_list_worker_jobs | List background jobs by status/queue |
moul_enqueue_job | Enqueue a new background job |
moul_cancel_job | Cancel a pending or retryable worker job |
moul_list_feature_flags | List all feature flags and gate rules |
moul_set_feature_flag | Create or update a feature flag |
moul_get_system_metrics | Fetch host CPU, Memory, Disk, and Load metrics |
moul_get_analytics_summary | Fetch visitor and request analytics totals |
moul_list_requests | Query recent HTTP request logs |
Custom Binary with MCP Support
When extending Moul as an embedded Go library with pkg/app, you can register custom domain MCP tools using moulApp.RegisterMCPTool(...). See examples/custom-binary-with-mcp/ for a complete runnable example demonstrating custom tools, route handlers, worker tasks, and dual-mode stdio/HTTP transport.
Outbound HTTP Webhooks
Moul collections support outbound HTTP webhooks registered per collection to enable integrations with external automation tools and services.
Event Triggers & Hook Lifecycles
*:before Hooks (create:before, update:before, delete:before): Executed synchronously before physical database operations. If the target webhook returns a non-2xx HTTP status code or fails to respond within 5s timeout, the record operation is rejected and an error is returned to the client.*:after Hooks (create:after, update:after, delete:after): Executed asynchronously in the background after successful database operations without delaying client response latency.- Wildcard & Shorthand Matching: Setting event
"create" matches both create:before and create:after. Setting "*", "", or "all" matches all events.
Payload & Signature Format
Outbound webhook POST requests include the following headers and payload:
Content-Type: application/jsonX-Moul-Event: Event trigger name (e.g. create:before)X-Moul-Webhook-ID: Unique webhook IDX-Moul-Timestamp: Event RFC3339 timestampX-Moul-Signature: HMAC-SHA256 hex digest of payload body computed using the configured secret key.
JSON Payload:
{
"event": "update:after",
"moul": "products",
"record": { "id": "prod-123", "name": "Widget Updated", "price": 49.99 },
"old_record": { "id": "prod-123", "name": "Widget", "price": 39.99 },
"timestamp": "2026-07-31T19:54:08Z"
}
Webhook Management Endpoints
GET /api/moul/:name/webhooks - List all webhooks for a collection.POST /api/moul/:name/webhooks - Create a new webhook for a collection.GET /api/moul/:name/webhooks/:id - Get webhook configuration by ID.PATCH /api/moul/:name/webhooks/:id - Update webhook configuration by ID.DELETE /api/moul/:name/webhooks/:id - Remove a webhook by ID.POST /api/moul/:name/webhooks/:id/test - Trigger a ping test payload (event: "ping") to verify webhook receiver connectivity.
Real-time SSE Subscriptions
Moul provides high-performance real-time record subscriptions over Server-Sent Events (SSE). Frontend applications can open an SSE connection to react to record changes (create, update, delete) in real time without polling.
Continuous Mid-Stream Authorization & Token Revocation
Realtime subscriptions feature continuous mid-stream authorization re-evaluation:
- Per-Event Rule Evaluation: Every record mutation (
create, update, delete) evaluates the collection's latest subscribeRule (or fallback viewRule) against the subscriber's auth context and the mutated record before dispatching frames.
- Transition-Aware State Sync: When a record update causes an item to no longer satisfy the subscriber's view rule (e.g., status changed from public to draft), the server emits a synthetic
delete event so client UIs immediately drop it.
- Immediate Disconnect on Revocation / Expiry: Subscribed clients track JWT credentials. If a token is revoked (via
POST /api/moul/:name/logout or db.RevokeToken) or expires during a connection, the server immediately emits an event: auth_expired frame and terminates the SSE connection.
- Slow-Consumer Eviction & Backpressure Defense: Each client stream operates on a bounded 128-event ring buffer with 5-second write deadlines on all flushes. If a client stalls their TCP window or cannot keep up, an
event: slow_consumer frame is emitted and the stream is cleanly closed to prevent memory exhaustion.
- Concurrent Connection Quotas: When rate limiting is enabled, active SSE streams are capped at 15 concurrent connections per authenticated user and 30 per IP (with customizable
maxConcurrent rate limiting rules), returning HTTP 429 Too Many Requests when limits are exceeded.
Subscribing to Collection Events
- Endpoint:
GET /api/moul/:name/subscribe
- Query Parameters:
event: Optional comma-separated list of event actions to filter (create, update, delete, or *). Default: *.id: Optional filter for a specific record ID.ticket: Optional short-lived single-use SSE ticket obtained via POST /api/realtime/ticket.
Security Note: Passing credentials in URL query strings (e.g. ?token= or ?admin_key=) is strictly disabled and returns HTTP 400 Bad Request to prevent credential exposure in reverse proxy logs, browser histories, and Referer headers. Always use Authorization: Bearer headers (via fetch() readable streams) or exchange for a short-lived single-use ticket.
JavaScript Client Examples
Option 1: Single-Use Ticket Exchange (for standard browser EventSource)
// 1. Request an opaque, 30-second single-use ticket
const { ticket } = await fetch('/api/realtime/ticket', {
method: 'POST',
headers: { 'Authorization': 'Bearer ' + userToken }
}).then(r => r.json());
// 2. Connect safely via EventSource using the single-use ticket
const evtSource = new EventSource(`/api/moul/posts/subscribe?ticket=${ticket}`);
evtSource.addEventListener('create', (e) => {
const data = JSON.parse(e.data);
console.log('New post created:', data.record);
});
Option 2: Direct Header Streaming via moul-live / fetch()
import { subscribeWithFetch } from '@moul-dev/ui'; // or ui/src/api/moul-live
const unsubscribe = subscribeWithFetch({
url: '/api/moul/posts/subscribe',
token: userToken, // Dispatched via standard Authorization: Bearer header
onMessage: (event) => {
console.log(`[${event.moul}] ${event.action}:`, event.record);
}
});
Global Subscriptions
To subscribe across all collections or specific multi-collections, use GET /api/moul/subscribe:
GET /api/moul/subscribe?mouls=posts,comments&ticket=tkt_...
Image Uploads
POST /api/upload stores the original and, for images, returns width, height, a thumbhash and thumbs with sm (256 px), md (1024 px) and lg (2048 px) renditions; a size that is not smaller than the original points at the original. Each rendition is cut from the next larger one with a Linear filter, at most MOUL_IMAGE_CONCURRENCY images are processed at once, and images above 40 megapixels are refused from their header, before anything is stored or decoded.
Set the file_thumbs_async setting to "true" to return as soon as the original is stored: the upload response then has width and height and every thumb pointing at the original, and the built-in GenerateImageSizes job makes the renditions and thumbhash and records them in the upload's meta.json (as listed by GET /api/upload).
Request Log & Visit Tracking
Every request outside the built-in excludes (docs, MCP, OAuth, the admin console, /api/visits, /api/requests, /api/workers) is logged to _requests (about 290 bytes per row), written in batches of up to 100 rows, one transaction per batch. The log is controlled by these _settings keys (PATCH /api/settings, applied immediately; POST /api/settings/reload picks up changes made by another process):
| Setting | Default | Description |
|---|
request_log_enabled | true | Turn the _requests log on or off. |
request_log_sample_rate | 1 | Fraction of eligible requests logged, from 0 to 1 (e.g. 0.1 logs one in ten). |
request_log_exclude_paths | "" | Comma-separated path prefixes never logged, in addition to the built-in excludes (e.g. /custom/poll,/api/health). |
request_log_retention_days | 30 | Days _requests rows are kept by the daily cleanup job. |
visit_retention_days | 30 | Days _visits rows are kept by the daily cleanup job. |
Visits. A request that brings a visit (moul_visit / moul_visitor cookie, or X-Visit-Token / X-Visitor-Token header) is attributed to it; known visits are cached in memory, so repeat requests run no SQL for tracking. A request that brings none starts a new visit only when the browser will keep the cookie (Sec-Fetch-Site: same-origin or none: the admin console, a page or link opened on the API's own origin). API clients, SSR, cross-origin fetches without credentials and probes never create _visits rows; their requests are logged with a NULL visit_id.
Token revocation checks answer from an in-memory copy of _revoked_tokens: a token revoked in this process (logout, db.RevokeToken) is rejected at once, and the copy is re-read every 60 seconds to honour revocations made by other processes (MCP stdio, CLI).
Worker Engine Architecture & Operations
- State Transitions: Jobs navigate states (
available -> executing -> completed / discarded). Historical execution records are retained for metric inspections.
- Failures & Exponential Backoffs: If a job handler returns an error or panics, the engine logs error details, increments
attempt, and reschedules execution with an exponential backoff ((attempt^2 * 10) + 10 seconds + jitter) until reaching max_attempts (default: 20), after which it transitions to discarded.
- Immediate Dispatch: Enqueuing a job via HTTP or Go API immediately signals the worker engine over an in-memory channel, executing available jobs without waiting for polling tickers.
- Atomic Claims: A worker claims its next job with a single
UPDATE … RETURNING statement, so claims never collide with concurrent writes and no job is claimed twice.
- Graceful Shutdown: On OS interrupt signals (
SIGINT/SIGTERM), worker fetching halts immediately while existing running task handlers are given a graceful timeout to finish execution cleanly.
Data Modeling & Field Types
Moul supports 11 dynamic field types for schema definitions with automatic SQLite storage mapping, OpenAPI 3.0 type hints, and runtime validation:
| Field Type | Description | SQLite Storage | Validation & Constraints | OpenAPI Type / Format |
|---|
text | Character string data | TEXT | Optional min and max character length | type: string, minLength, maxLength |
number | Floating point or integer numeric values | NUMERIC | Optional min and max numeric bounds | type: number, minimum, maximum |
bool | Boolean flag | INTEGER (1/0) | Validates boolean (true/false, 1/0, "true"/"false") | type: boolean |
date | Date string | TEXT | Enforces YYYY-MM-DD ISO format | type: string, format: date |
datetime | Date and time timestamp string | TEXT | Enforces ISO 8601 / RFC 3339 format | type: string, format: date-time |
json | Arbitrary JSON object or array structure | TEXT | Validates valid JSON syntax; unmarshals back to JSON | type: object |
url | Web address link string | |
Encrypted Fields (cloak)
The cloak field type encrypts sensitive data (PII, SSNs, credit cards, secrets) at the application layer before writing to SQLite:
- Encryption Algorithm: AES-256-GCM with a version byte and random 12-byte nonce (
[0x01][nonce][ciphertext + tag]). Keys are derived via HKDF-SHA256 from MOUL_ENCRYPTION_KEY.
- Masking by Default: API responses mask cloak fields (
•••• + last 4 chars, or •••• if <= 4 chars). Admin callers always receive unmasked decrypted values. Non-admin users can request unmasked plaintext with ?reveal=field1,field2 (or ?reveal=all).
- Searchable Blind Indexes: Setting
searchable: true automatically provisions an indexed companion column (<fieldName>Hash TEXT) containing an HMAC-SHA256 blind index. Allows exact equality queries (field = 'value' or field != 'value') while range/pattern queries (~, >, etc.) are securely rejected. Companion hash columns are never exposed in API responses.
Access Rules & Filters Syntax
Each Moul (table) supports five HCL-like expression rules evaluated on client API requests:
listRule - Restricts records returned in collection list queries.viewRule - Controls access to view a single record by ID.createRule - Validates fields/permissions before record insertion.updateRule - Validates current record fields and incoming values before update.deleteRule - Validates permissions before record deletion.
Rule Expression Syntax Reference
- Request Context Variables:
@request.auth.id, @request.body.fieldName, @request.headers.header_name, @request.query.paramName, @request.method
- Operators:
=, !=, >, >=, <, <=, ~ (LIKE/contains), !~ (NOT LIKE)
- Wildcard Array Modifiers:
?= (e.g. allowedUsers.id ?= @request.auth.id)
- Field Modifiers:
:lower, :length, :isset, :changed, :each
- Helper Functions:
geoDistance(lonA, latA, lonB, latB), strftime(format, timeVal)
- Cross-Collection Join Queries:
@collection.userRoles.userId = @request.auth.id && @collection.userRoles.role = 'admin'
Rules are written by admins and may use every construct above. Client ?filter= expressions use the same syntax with the restrictions listed under Security Defaults.
Security Defaults
Moul ships secure by default. The behaviors below apply to every caller that is neither a root user (_rootUsers token) nor the admin key.
- Admin surface: schemas (
/api/moul, /api/moul/:name, webhooks, email templates, export/import), settings, file listing and deletion, workers, retry-jobs, visits, requests, feature flag management, system metrics, the rules sandbox and MCP accept only a root user token or the admin key, and only they get collection schemas in /openapi.json / /openapi.yml (others get the static engine spec). Guests get 401 (with the OAuth discovery challenge), members of other auth collections 403. Members keep POST /api/upload and POST /api/feature-flags/:key/eval (identity attributes and a percentage rollout's attribute are pinned to the caller and org_id is dropped; other attributes such as groups and role remain client-asserted, so flags are not an authorization mechanism).
- Client IP: forwarding headers are honored only from trusted proxies. By default only loopback is trusted (a reverse proxy or tunnel agent on the same host). A proxy in another container or on the LAN needs
MOUL_TRUSTED_PROXIES (IPs, CIDRs, loopback, private for every private network, or none); use private only when no client can reach Moul from a private address (NAT setups such as Docker Desktop or Kubernetes SNAT). Optionally set MOUL_CLIENT_IP_HEADER (e.g. CF-Connecting-IP). Rate limits, the root IP allowlist, OTP throttling, the request log and visits all use this address.
- Hidden records answer like missing ones:
GET, PATCH and DELETE on /api/moul/:name/records/:id check the viewRule first; a record the caller may not view answers 404 Record not found, byte for byte the answer for an unknown id (401 for both when the presented bearer token was rejected, so clients still notice an expired session). Relation targets of a create are validated only after the createRule passes.
- Filters, sorts and search: the
listRule and ?filter= bind their values under separate placeholders, so a filter can never rewrite the rule. Client filters may not use @collection, and neither filters nor sorts may reference secret columns (passwordHash, tokenKey, OTP and reset columns, oauthProviders, and the email of auth collections). Relation paths (author.name = 'x') only match target records the caller may list; sorting through a relation requires a publicly listable target. Every column, including relation targets, is validated against the schema before it reaches SQL. skips emails, and the cursor is read under the caller's .
Deployment Information
Single-Binary Production Build
moul compiles into a single, self-contained binary containing the HTTP engine, worker processor, embedded web docs, and CGO-free SQLite database driver.
To build the production binary with stripped debug symbols and version metadata:
make build
# Creates executable at bin/moul
Versioning & Releases
Moul and @moul-dev/ui share one CalVer scheme, YYYY.MM.MICRO: the UTC year, the UTC month without zero padding, and a release counter that starts at 0 each month. Release tags add a leading v, so the first release of November 2026 is v2026.11.0 and the next is v2026.11.1. Every tag is valid SemVer.
make next-version # Prints the next tag, e.g. v2026.11.0
make release-tag # Tags HEAD with it; publish with: git push origin <tag>
Pushing the tag runs .github/workflows/release.yml. It rejects any tag that is not vYYYY.MM.MICRO, then builds moul and moul-ctl for macOS and Linux and publishes them as a GitHub release. The tags from v2026.07 to v2026.09 predate the scheme.
Production Environment Variables
Set the following environment variables on your production server or container:
| Variable | Required | Description | Example |
|---|
MOUL_ENV | Yes | Application environment mode | production |
MOUL_ADMIN_KEY | Yes | Master administrative secret key | super-secret-admin-key-9988 |
MOUL_JWT_SECRET | Yes | Secret key for signing JWT tokens | jwt-secret-key-production-3344 |
MOUL_PORT | No | HTTP listening port (default: 8090) | 8090 |
MOUL_API_PREFIX | No | Base URL path prefix for API endpoints (default: /api, use "" for root) | /v1 or "" |
MOUL_PUBLIC_URL | No | Base public URL for email links (default: http://localhost:8090) | https://api.myapp.com |
MOUL_DB_PATH | No | Path to SQLite database file | /var/lib/moul/moul.db |
MOUL_IMAGE_CONCURRENCY | No | How many uploaded images are decoded and resized at once (default: max(1, CPU count / 2)); further uploads wait | 2 |
MOUL_DB_MAX_CONNS | No | SQLite connection pool size; every connection is opened with WAL, busy_timeout=5000, synchronous=NORMAL and foreign_keys=ON, and idle connections are kept (default: max(4, CPU count)) |
Litestream Automated S3 Backup & Disaster Recovery
moul includes built-in Litestream replication directly in the binary for real-time, point-in-time SQLite replication to S3-compatible cloud storage (AWS S3, MinIO, Cloudflare R2, DigitalOcean Spaces).
Enabling Replication in Production
To enable background replication, set the following environment variables before running moul start:
LITESTREAM_ENABLED=true
LITESTREAM_S3_BUCKET=my-moul-backups
LITESTREAM_ACCESS_KEY_ID=YOUR_AWS_OR_S3_ACCESS_KEY
LITESTREAM_SECRET_ACCESS_KEY=YOUR_AWS_OR_S3_SECRET_KEY
LITESTREAM_REGION=us-east-1
# Optional for S3-compatible services (MinIO, R2, Cloudflare, DigitalOcean):
LITESTREAM_S3_ENDPOINT=https://s3.us-east-1.amazonaws.com
LITESTREAM_S3_FORCE_PATH_STYLE=false
When enabled, moul automatically streams SQLite Write-Ahead Log (WAL) changes to S3 asynchronously with zero downtime.
Database Disaster Recovery / Restore
To restore a database state from S3 backup onto a new server:
# Run restore command using the same Litestream S3 env configuration
moul restore --db /var/lib/moul/moul.db
Systemd Service Deployment Example
For VPS deployments (Ubuntu/Debian), create a systemd service at /etc/systemd/system/moul.service:
[Unit]
Description=Moul Dynamic Database & Engine
After=network.target
[Service]
Type=simple
User=moul
Group=moul
WorkingDirectory=/var/lib/moul
ExecStart=/usr/local/bin/moul start
Restart=always
RestartSec=5
Environment=MOUL_ENV=production
Environment=MOUL_ADMIN_KEY=your-production-admin-key
Environment=MOUL_JWT_SECRET=your-production-jwt-secret
Environment=MOUL_DB_PATH=/var/lib/moul/moul.db
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
Enable and start the service:
sudo systemctl daemon-reload
sudo systemctl enable --now moul
Production Deployment with LXD, Tailscale & Cloudflare Tunnel
For hardened enterprise production deployments on Ubuntu Server 26.04 LTS, moul can be deployed in an unprivileged LXD system container combined with Tailscale for private host management and Cloudflare Tunnel (cloudflared) for zero-trust public HTTPS ingress.
Key features of this deployment architecture:
- Zero Open Public Inbound Ports: Firewall drops public SSH (22) and HTTP/HTTPS (80/443).
- Tailscale Host Operations: SSH and LXD container administration (
lxc exec) are isolated to the tailscale0 network interface.
- Cloudflare Ingress: Public API clients and Admin TUI connect securely via Cloudflare Tunnel over HTTPS.
- Litestream Replication: Built-in point-in-time SQLite replication to S3/R2 storage.
📖 Complete Guide: docs/deployment-lxd-tailscale-cloudflare.md
