ObserveCraft is an open-source, self-hosted observability platform: logs, metrics, traces, dashboards, monitors, incidents and CI visibility, built on OpenTelemetry and ClickHouse.
One governed MCP server connects any AI agent to a finite action catalog, enforced on-host with pack trust, policy gates, approvals, and an audit trail.
Open-source, self-hosted observability platform: logs, metrics, traces, dashboards, monitors, incidents and CI visibility in one place, built on OpenTelemetry and ClickHouse.
Open-source, self-hosted observability platform: logs, metrics, traces, dashboards, monitors, incidents and CI
visibility in one place, built on OpenTelemetry and ClickHouse.
One place for your telemetry. Search logs, explore traces, chart metrics, alert on all of them and run incidents
without stitching five tools together.
Open standards in, your storage underneath. Send data with OpenTelemetry or Vector; it lands in ClickHouse
tables you own and can query directly.
Fast and durable. A Rust (axum) API over ClickHouse columnar storage. The intake writes what it accepts to
Kafka before answering, so processing restarts and storage outages don't lose data, and live tail streams from it.
Enterprise-ready auth from day one. Password + MFA, SAML 2.0, OpenID Connect, RBAC, application keys and an
audit trail.
Multi-tenant. Organizations keep their users, SSO, content and telemetry apart; one account can belong to
several organizations and switch between them.
Self-hosted and MIT-licensed. No per-host or per-GB pricing, and no data leaving your network.
Features
Area
Highlights
📜
Logs
Query language with facets, histogram, live tail (SSE), anomaly insights, list/timeseries/top-list/table/tree-map/pie visualizations, trace ↔ log links
🔀
Splunk migration
Translate SPL searches step by step (filters, stats, timechart, top, macros) and import exported alerts and dashboards as log monitors and dashboards after a review
🗂️
Log indexes
Per-org indexes (Datadog model): filters route each log to the first matching index, each with its own retention, daily quota and sampled exclusion filters; Live Tail and archives still get every log
🗄️
Log archives
Hourly Parquet archives in your own S3-compatible bucket, Archive Search with DuckDB (no rehydration needed), rehydration back into the explorer with scan limits
📤
Log forwarding
Custom destinations: logs matching a query, after pipelines, forwarded to an HTTP endpoint, Splunk HEC or Elasticsearch with tag filtering; Flex Logs tier for longer index retention; all Logs Configuration resources managed with Datadog's Terraform provider (datadog_logs_*)
🧭
APM
Service catalog, endpoints, recommendations (N+1 queries, error hot spots), deployments, service pages with latency/error breakdowns
🌐
Cloud Network Monitoring
Traffic between clients and servers grouped by any tag (service, availability zone, namespace, pod, host, IP, port): volume, throughput, TCP retransmits, latency and jitter, connections; DNS view (requests, response time, timeouts, SERVFAIL, NXDOMAIN by domain); summary graphs, client/server facets, flow side panel, network map
🔥
Continuous Profiler
Datadog profiler uploads: CPU and memory per service over time, merged flame graphs with endpoint and package breakdowns, profile list, A/B comparison with differences coloured, CPU-burst insights with review status
🔎
Trace explorer
Span and trace search, facets, group-by analytics, flame graph, waterfall, span list, service map, span ↔ host metrics, logs and processes
📈
Metrics
Explorer and summary, query engine (avg:system.cpu.user{env:prod} by {host}), gauges, counters, histograms, exponential histograms and summaries (http.server.request.duration.p95), span-derived trace.* metrics
🧩
Dashboards
Drag-and-drop dashboards, lists, starring, built-in host and APM dashboards
🖥️
Infrastructure
Host list and map, host details with metrics, processes, pods, containers, spans and logs; Processes explorer; Event Management (All Events)
☸️
Containers
Kubernetes Overview and Explorer (pods, workloads, nodes, network, policies, storage, access control, CRDs, containers, container images) with facets, group-by, map and side panels (YAML, logs, APM, metrics, events, monitors); Resource Utilization; ECS Explorer (tasks, task definitions, services, clusters). Collector manifests in deploy/examples/kubernetes, ECS (FireLens + ADOT) in deploy/examples/ecs
🚨
Monitors
Metric, APM, log and change monitors; warn/alert thresholds with recovery; backtesting; downtimes; notifications to webhooks (HMAC-signed), Slack and email with retries; quality checks
🧯
Incidents
Declare, triage, timeline, impacts, follow-ups and analytics
🚀
Software Delivery
CI pipeline explorer, flaky-test management, DORA metrics from OpenTelemetry CI/CD semantic conventions; GitLab CI pipelines pulled through GitLab's API (no inbound access needed)
🔐
Identity & tenancy
Organizations with an instance admin console, one account across orgs, password + TOTP MFA, SAML 2.0, OIDC, roles, teams, SAML group mappings, application and ingest API keys, audit trail
APM: services and recommendations
Trace explorer
Monitors
Infrastructure
CI/CD pipelines
Quickstart
Requirements: Docker (Docker Desktop, OrbStack, Colima, …), Rust ≥ 1.89, Node.js ≥ 20, and libxmlsec1 + libxslt
for SAML (brew install libxmlsec1 libxslt on macOS, apt install libxmlsec1-dev libxml2-dev libxslt1-dev pkg-config
on Debian/Ubuntu).
git clone https://github.com/observecraft/observecraft.git
cd observecraft
(cd web && npm install)
make up # ClickHouse, Postgres, Kafka, Valkey, Vector, OTel Collector, RustFS (S3), synthetic logs
make server # API on :8080 (applies migrations on startup)
make intake # telemetry intake on :4318 (OTLP/HTTP + JSON logs, API-key authenticated)
make tracegen # demo traces, logs, metrics and CI pipelines, sent through the intake
make web # UI → http://localhost:5173
The demo generator also exports fictional Kubernetes telemetry for shop-prod-eks and
shop-staging-eks: nodes, pods, deployments, ReplicaSets, StatefulSets, DaemonSets, a completed
CronJob job and a healthy control plane. It sends compatibility metrics through the authenticated
OTLP intake every 10 seconds, so integration dashboards use the same storage/query path as real
telemetry. On startup it replays 30 minutes at one-minute resolution; set
OC_K8S_BACKFILL_MINUTES=0 to disable replay (maximum: 60). Resources carry
telemetry.synthetic=true. The generator also sends kubelet service-check statuses every 10 seconds and workload
reconciliation events every minute. It does not produce Kubernetes object snapshots;
connect a real collector to populate the Kubernetes Explorer inventory.
Dashboard metric panels share a query plan. In live mode, a single authenticated
POST /api/v1/query/dashboard/live SSE connection returns the initial history and then
replaces the last two aligned buckets every 10 seconds. Compatible gauge/counter queries
share telemetry scans; distributions and span-derived metrics retain their own evaluators.
Identical subscriptions within an organization share the live plan and history. A paused view
uses POST /api/v1/query/dashboard once. Logs, events and distribution heatmaps use their
respective data-source endpoints. Shared metric frames include units from telemetry; display
conversions leave formula inputs unchanged. exclude_null() removes missing group tags while
retaining valid zero values. Event charts use the server's status histogram; integration tables
join formula columns by series group. Bundled notes contain ObserveCraft-authored guidance and
illustrations rather than upstream marketing copy or artwork.
The dashboard weighted() modifier accepts gauge sums and weights transient series using their
observed regular submission cadence, including incomplete live buckets. Without enough samples
to infer a cadence it keeps the unweighted sum. This is inferred from telemetry, not configured
submission-interval metadata. top() supports bounded ranking by a requested aggregation.
Forecasts, historical-shift formulas, and unsupported panel types remain explicit errors.
On first start the server logs a one-time link to set the admin password:
No one can sign in yet. Set the admin password for … : http://localhost:5173/login/link/…
That account (OC_INSTANCE_ADMIN_EMAIL) is also the instance admin: account menu → Instance Admin creates
organizations and invites their first admins. The demo org archives its logs to the local RustFS bucket
(console on http://localhost:9101).
Everything in containers:docker compose -f deploy/docker-compose.yml --profile app up -d runs the published
image (ghcr.io/observecraft/observecraft, set OC_VERSION to pin a release; built from source when it can't be
pulled) with the API, the web app (served on :8080), the intake (:4318) and the demo generator. Add --profile tls
(with OC_DOMAIN and OC_INTAKE_DOMAIN) for automatic HTTPS through Caddy.
UI only, no backend:make web-mock runs the UI against in-browser mocks.
Try SSO locally:docker compose -f deploy/docker-compose.yml --profile sso up -d starts a SAML IdP
(SimpleSAMLphp on :8081, user user1 / user1pass) and an OIDC provider (issuer http://localhost:8082/default).
Configure them under Organization Settings → Login Methods.
Send your own data
Create an API key under Organization Settings → API Keys. Everything sent with it belongs to that organization.
The intake accepts what Datadog accepts, so you can keep Datadog's open-source agents and libraries:
OTLP, over HTTP (:4318) and gRPC (:4317), for traces, metrics and logs;
Datadog's API and Agent formats: metrics, distributions, logs, APM traces, service checks, and the Cluster
Agent's Kubernetes manifests.
OpenFeature flag evaluation (OFREP, /ofrep/v1/evaluate/flags): any OpenFeature SDK with the OFREP provider
evaluates the flags managed under Software Delivery → Feature Flags.
The Datadog Agent (official Helm chart or Operator) and Datadog's tracing libraries (dd-trace-*) and DogStatsD
clients need only a new URL and key. Fluent Bit, Fluentd, Vector and Logstash work through their Datadog outputs.
For Kubernetes clusters and hosts, run the Datadog Agent (its Helm chart or Operator) pointed at the intake, or, for
an OpenTelemetry-only setup, the observecraft-agent Helm chart (the official OpenTelemetry Collector, configured).
docs/sending-data.md has the setup for every sender: the Datadog Agent and tracers, DogStatsD,
OpenTelemetry, Fluent Bit, Fluentd, Vector,
Logstash, Filebeat, Prometheus and syslog.
Configuration can be managed as code with Datadog's Terraform provider and API clients pointed at the server
(api_url, an application key ocak_…, optionally an API key ocik_…); support grows resource by resource, see
docs/datadog-api.md.
Logs written inside a trace (OTLP logs carry trace_id/span_id) link to it both ways: the span's Logs tab and the
log's Trace tab.
Deploy on Kubernetes
The platform is one Helm chart, published with every release as an OCI artifact next to the image
(ghcr.io/observecraft/observecraft, amd64 and arm64). Pick a version from the
releases:
For an OpenTelemetry-only setup, use the observecraft-agent chart instead. It runs the official OpenTelemetry
Collector as a node agent and a cluster agent.
For production, values-ha.yaml runs ClickHouse sharded and replicated with
Keeper (Altinity operator), Postgres as a CloudNativePG cluster with failover, three Kafka brokers (replication 3), and autoscaled,
disruption-budgeted services. Or point the chart at your own databases (clickhouse.mode=external,
postgres.mode=external) and cache (cache.mode=external: ElastiCache, Memorystore or any Valkey/Redis-protocol
server).
The chart generates and keeps its secrets (or uses secrets.existingSecret), runs everything as non-root with
read-only root filesystems, and restricts traffic between components with NetworkPolicies. Every server replica can
start at once: migrations run under a Postgres lock, and monitors and archiving run on one replica at a time.
ClickHouse gets a per-organization quota (clickhouse.quota). helm test observecraft checks the install.
Upgrade with helm upgrade observecraft oci://ghcr.io/observecraft/charts/observecraft --version <new> --reuse-values;
migrations run on startup. Read the release notes first: before 1.0, a minor version may change chart values.
Images and charts are signed and come with an SBOM and build provenance; SECURITY.md
shows how to verify them. Every commit on main is also published as :main and :sha-<commit> images, for staging.
Architecture
flowchart LR
subgraph Sources
A[Apps & services<br/>OpenTelemetry SDKs]
H[Clusters & hosts<br/>ObserveCraft Agent]
C[CI pipelines<br/>OTel CI/CD spans]
end
D[Datadog Agent · Fluent Bit<br/>Vector · Logstash]
A & H & C -->|OTLP HTTP/gRPC + API key| IN[oc-intake<br/>auth · rate limits]
D -->|Datadog logs / series API| IN
IN -->|acks=all · keyed by org| K[[Kafka<br/>durable buffer · 72 h]]
K -->|OTLP| COL[OTel Collector]
K -->|JSON logs| VEC[Vector aggregator]
COL -->|logs| VEC
COL --> CH[(ClickHouse<br/>logs · traces · metrics · CI<br/>per org)]
VEC --> CH
VEC -->|live tail| K
API[observecraft-server<br/>Rust · axum] --> CH
API --> PG[(Postgres<br/>orgs · accounts · dashboards<br/>monitors · incidents)]
API -->|live tail| K
API <-->|shared results| VK[(Valkey<br/>cache)]
API <-->|hourly Parquet| S3[(S3-compatible bucket<br/>log archives)]
API -.->|DuckDB| S3
API -->|webhook · Slack · email| OUT((Notifications))
UI[Web app<br/>React · TypeScript] <-->|JSON · SSE| API
Every telemetry row carries its organization: the intake resolves the API key and writes the org as a Kafka record
header, the Collector stamps the resource with oc.org_id (replacing anything a sender put there), and every query
filters on it. The intake answers a sender once Kafka has its data on all in-sync replicas; the Collector and Vector
commit their Kafka offsets only after storing (see ADR-0004).
Path
What
web/
React + Vite + TypeScript UI. src/api/*/client.ts is the only data-access layer.
ClickHouse (telemetry) and Postgres (metadata) schemas.
deploy/
Docker Compose stack, Helm charts (helm/observecraft, helm/observecraft-agent), Vector aggregator (vector/normalize.vrl, with unit tests) + load generator, OTel Collector config (shared by Compose and Helm), RustFS, Caddy (TLS), Kubernetes and ECS examples (examples/).
API contract
The OpenAPI document is generated from the Rust handlers (utoipa), and the web's TypeScript types are generated from
it: run make openapi after changing an endpoint. web/src/api/http.ts checks paths, bodies and responses against
web/src/api/generated/schema.d.ts at compile time. The live spec is served at GET /api/openapi.json.
Configuration
The server reads its settings from environment variables; see .env.example for the full list. The
most important ones:
Limits on each telemetry query of an organization (OC_QUERY_TIMEOUT_SECONDS / OC_QUERY_MAX_MEMORY_BYTES still work); queries also send the org as ClickHouse quota_key for per-org quotas. A query stopped at a limit gets 503 with what to change
ClickHouse queries of one organization running at once on each server replica (0 = no limit); more wait up to the timeout, then the API answers 429 ("retry shortly"). Background jobs (metric rollups, monitors, schedulers) are exempt
Kafka bootstrap brokers (host:port,…). The server reads live tail from it; the intake (oc-intake) writes telemetry to it, and forwards over HTTP to the Collector and Vector when it is unset
OC_CACHE_URL
—
Required. Shared cache of all server replicas: Valkey (or another Redis-protocol server), redis://[user:password@]host:6379/0 or rediss://… for TLS. The server exits at startup when it is unset or unreachable for 30 s; later outages only cost cache hits
OC_DATADOG_API_ALLOW_APP_KEY_CREATION
false
Lets the Datadog-compatible API create application keys when the caller is itself an application key (Terraform's datadog_application_key). Off by default so a leaked key can't create keys that outlive its revocation; see docs/datadog-api.md
Largest value stored (after compression); how long replicas wait for another replica's load of the same entry
OC_INVENTORY_CACHE_MAX_OBJECTS
500000
Kubernetes objects the in-process inventory cache keeps over all orgs
OC_PUBLIC_URL
http://localhost:5173
Origin users open the app on (SSO redirects, login links, notification links)
OC_AWS_PRINCIPAL_ARN
unset
The AWS IAM role the server assumes customers' AWS integration roles with; their roles trust it (Integrations → Amazon Web Services)
OC_GCP_PRINCIPAL_EMAIL
unset
The GCP service account the server impersonates customers' service accounts with (the STS delegate); they grant it Service Account Token Creator
OC_CLOUD_CRAWLERS
aws,azure,gcp
The cloud integrations whose metrics the server crawls (CloudWatch, Azure Monitor, Cloud Monitoring); needs OC_KAFKA_BROKERS
OC_CLOUD_CRAWL_INTERVAL_SECS
600
How often each account's region, subscription or project is read
OC_CLOUD_CRAWL_CONCURRENCY
8
Crawl jobs one server runs at a time per cloud (add servers to crawl more)
OC_SECRET_KEY
—
Required in production. Encrypts MFA seeds, OIDC secrets, archive and channel credentials (openssl rand -base64 48)
OC_INSTANCE_ADMIN_EMAIL
—
Account that creates and manages organizations (becomes an admin of the install's org)
OC_RETENTION_LOGS_DAYS / _TRACES_DAYS
15 / 15
Defaults for a new org's first log index and its span retention (orgs change them in Log Indexes / Data Retention)
OC_RETENTION_LOGS_MAX_DAYS / _TRACES_MAX_DAYS
365 / 90
Most the instance admin can grant an org (Instance Admin → Retention limits)
OC_RETENTION_METRICS_DAYS / _CI_DAYS
30 / 30
Install-wide metric and CI retention
OC_RAW_LOGS_HOURS
72
Every log stays this long unindexed, as the source of archives
OC_LOGPROC_ADDR / OC_INTERNAL_TOKEN
0.0.0.0:8090 / —
Internal listener: the log indexer Vector posts to, and the server's own metrics (GET /metrics, Prometheus text: oc_cache_*). Keep it off the public network; the token is optional
OC_SMTP_URL / OC_SMTP_FROM
—
SMTP server for monitor emails (smtps://user:pass@host:465)
OC_ASSIST_PROVIDER
—
The install's default model for Assist: anthropic (the Messages API) or openai (any OpenAI-compatible chat completions endpoint: OpenAI, Azure OpenAI, vLLM, LiteLLM, Ollama). Each organization can set its own in Organization Settings → Assist AI (key sealed), which replaces this one for it. Unset and no organization provider: Assist says it isn't set up
The provider's key (required for anthropic), model (required for openai) and URL (required for openai, e.g. https://api.openai.com/v1)
OC_ASSIST_ALLOW_PRIVATE
false
Lets organizations point Assist at private addresses (a model on the install's network); leave off on multi-tenant installs
OC_ASSIST_FALLBACKS / OC_ASSIST_MAX_TOKENS
true / 64000 (anthropic), 4096 (openai)
Anthropic's server-side fallback when the model declines a request; the most a model turn writes
OC_SYNTHETICS_ALLOW_PRIVATE
false
Let synthetic tests reach loopback, private, link-local (cloud metadata) and other internal addresses
OC_DATA_DIR
.oc-data
Local cache for Archive Search and rehydration
OC_COOKIE_SECURE
false
Set true when served over HTTPS
OC_SESSION_HOURS
12
Idle session lifetime (sessions also end 7 days after sign-in)
OC_WEB_DIST
—
Serve the built web app from the API (production-style)
OC_LOG_FORWARD_ALLOW_PRIVATE
false
Custom destinations may forward logs to private and loopback addresses
OC_KEV_URL
CISA's catalog
Known Exploited Vulnerabilities JSON the Security research feed reads every six hours (off without internet access)
OC_SEED_DEMO
true
Fictional demo users, teams, monitors and incidents, a demo ingest key and archive. Set false in production.
The intake (oc-intake) has its own settings: OC_INTAKE_ADDR (HTTP, :4318), OC_INTAKE_GRPC_ADDR (OTLP/gRPC,
:4317), OC_INTAKE_COLLECTOR_URL, OC_INTAKE_LOGS_URL, OC_INTAKE_AUTOSCALING_URL (the server's internal listener for the
Datadog Cluster Agent's external recommender calls), OC_INTAKE_MAX_BODY_MB (16), and per-organization limits
OC_INTAKE_ORG_RPS (500 requests/s) and OC_INTAKE_ORG_MBPS (50 MB/s).
Security
Tenancy: every Postgres row and every ClickHouse telemetry row carries its organization; a request's org comes
only from its session or application key, a sender's org only from its ingest API key — never from client input.
An integration test (server/crates/oc-api/tests/tenancy.rs) checks isolation end to end.
Accounts across orgs: links issued by one org's admins can never set the password of an account that also
belongs to another org; sessions started with SSO stay in that org.
Queries: all search values and tag keys are bound parameters (oc-query::build_where), never interpolated into
SQL.
Authentication: Argon2id passwords, TOTP MFA, account lockout, signed SAML responses (SHA-256+, replay
protection), OIDC with PKCE and nonce, HttpOnly SameSite cookies with origin checks on writes.
Roles: Admin, Standard and Read Only, enforced by the API.
Found a vulnerability? Please report it privately; see CONTRIBUTING.md.
Project status
ObserveCraft is under active development and not yet production-ready: APIs and schemas may still change between
releases.
Area
Status
Ingest (OTLP HTTP/gRPC, Datadog logs and metrics APIs), API keys, organizations
Cloud SIEM: content packs (CloudTrail, Okta, GitHub, Kubernetes audit, web servers, …), log detection rules and suppressions (Terraform datadog_security_monitoring_rule / _suppression), signals and triage; Security overview, inbox and research feed (CISA KEV)
✅ Live (ClickHouse, Postgres)
Cloud Security: host and container image vulnerabilities from the Agent's SBOMs, Kubernetes and host misconfigurations, inventory, infrastructure packages, CVE explorer
✅ Live (Postgres, ClickHouse)
App & API Protection: security traces from the tracers' AppSec spans, attackers and users, API endpoints and findings (OWASP API Top 10), In-App WAF custom rules, exclusion filters and policies (Terraform datadog_appsec_waf_custom_rule / _exclusion_filter), denylist and detection rules
✅ Live (ClickHouse, Postgres); not yet pushed to tracers
Workload Protection: the Agent's runtime security events, agent policies and rules (Terraform datadog_csm_threats_policy / _agent_rule), Agents and coverage, workload_security detection rules and signals, workload_activity findings, MITRE ATT&CK coverage
✅ Live (ClickHouse, Postgres); policies reach Agents as downloaded files
AI Guard: the tracers' AI Guard SDK evaluate endpoint with our own evaluator (prompt injection, jailbreak, prompt extraction, exfiltration, destructive tool calls, sensitive data), Investigate with the conversation timeline, Signals, Discover (AI services and model providers from LLM spans), Playground, per-service blocking
✅ Live (ClickHouse, Postgres)
Live debugger
🗺️ Planned
Contributing
Contributions are very welcome! Read CONTRIBUTING.md for the development setup, coding guidelines
and pull-request process. A quick check before you push:
make test # Rust tests + web typecheck
make openapi # if you changed an API endpoint
Support ObserveCraft
Sponsor @andreimerfu, ObserveCraft's maintainer, to help fund
open-source development, CI and releases, and documentation.
You can also support the project by reporting issues, improving documentation and contributing fixes.
