synthkit

Get a realistic, Grafana-visible metrics, logs, and traces environment from one YAML blueprint — without deploying a real application.

Documentation — installation, blueprint reference and the signal catalog.

Composable synthetic-telemetry generator for Grafana Cloud. Declare a model of the infrastructure and applications you want in one YAML blueprint — synthkit does not deploy those applications or infrastructure — and it emits structurally-correct synthetic metrics, traces, and logs (plus optional Faro/RUM) with the REAL metric/label/field names of each technology it models: EKS + the k8s-monitoring substrate, EC2/ALB/NAT-gateway/EBS/ S3 CloudWatch families, RDS/ElastiCache, Database Observability (MySQL/Postgres), Azure/GCP CSP, Cloudflare, Synthetic Monitoring, Fleet Management, and correlated request workloads — web_service (a single service) and app (a declared multi-service GRAPH whose nodes emit custom metrics/logs/ spans via a telemetry DSL) — sharing one end-to-end correlation ID per request across the graph.

• Catalog: isolated construct/workload modules, each with a verified signal contract (SIGNALS.md).
• Composition: a blueprint is one deletable YAML file; constructs know nothing about it. See ARCHITECTURE.md.

The runtime control plane

Every deployment serves an operator UI at /control/ui — live per-blueprint emission rates, a per-construct cardinality inventory, and on-demand failure injection.

Curated incident scenarios fire correlated failures across constructs — a bad deploy, a node-failure cascade, a retrieval meltdown — so a dashboard or alert can be demonstrated against a real signal.

The X-ray view accounts for every series before it leaves the process: metric names, label keys and cardinality per construct.

Start here

Follow the Quick Start for one focused blueprint: inspect its inventory without credentials, configure your destination, run preflight, then verify metrics, logs and traces. The installation guide covers source builds and prebuilt containers.

Open the diagram at full size.

A source build needs the frontend assets built before the Go binary to include the operator UI. The Quick Start includes both steps; published containers already include the UI.

For other starting points:

• Understand the model — blueprints, workloads and signal destinations.
• Write a blueprint — adapt an existing example.
• Contribute — developer setup, agent workflows and required checks.

LLM-assisted setup (Claude Code / Codex / OpenCode)

synthkit ships agent skills for deployment and operation. In Claude Code, open this repo and enter /initial-setup (also /verify-deployment, /create-blueprint, /setup-fleet-management) in the chat input. To install as a plugin from anywhere, enter these in Claude Code chat; they are not shell commands:

/plugin marketplace add rknightion/synthkit
/plugin install synthkit@synthkit      # → /synthkit:initial-setup, etc.

The same skills work in Codex (.agents/skills/) and OpenCode (reads .claude/skills/). Skills are authored once under plugins/synthkit/skills/; .claude/skills and .agents/skills are symlinks kept in sync by just skills-sync (verified by just skills-check).

Synthetic Monitoring — two-phase startup

Blueprints with synthetic_monitoring blocks use a snapshot-bound one-shot control-plane step. Start the emitter once to write the private snapshot, preview/apply the version-matched Docker job, then restart the emitter to activate the matching registration:

docker compose up -d synthkit
docker compose --profile sm-provision run --rm sm-provision

Review the printed preview plan. Only then authorise the remote changes and restart the emitter:

SM_PROVISION_APPLY=true docker compose --profile sm-provision run --rm sm-provision
docker compose restart synthkit

The first v2 activation-format upgrade also needs this explicit apply/restart. Later release-only bumps retain activation when specification, target and schema match; the write tool must still match the current snapshot version. The standalone in-cluster tool supports a dedicated named SM Lease and ConfigMap without shared disk. Helm also supports named SM state with HA; select a matching engine image (at least 1.6.0-rc.176). See Synthetic Monitoring.

Preview is the default and makes no remote mutation. DRY_RUN does not authorise the provisioner. Foreign collisions and ambiguous/pending operations fail closed; ordinary apply never deletes. Removal is the separate, previewed sm-provision cleanup (see Synthetic Monitoring). Probe name, region and paired optional latitude/longitude resolve once for both planes. Omitted coordinates retain Frankfurt; all checks sharing a probe name must agree on its location.

Authoring a blueprint

Copy blueprints/k8s-minimal.yaml, rename it, and declare the environments, clusters, databases, and synthetic workloads you want to model. This is a telemetry model, not a deployment manifest: synthkit does not create pods, databases, or application code. The schema is documented in ARCHITECTURE.md §3; unknown constructs or fields fail loudly at load. Deleting your blueprint file removes its telemetry and affects nothing else.

A fresh synthkit process selects no blueprints and emits no synthetic telemetry. Set BLUEPRINT_NAMES to one or more exact names before starting (for example, BLUEPRINT_NAMES=otlp-native); BLUEPRINT_NAMES=* is the explicit all-catalog opt-in.

name: mine
environments:
  - name: prod
    cloud:   { provider: aws, account_id: "210987654321", region: eu-west-1, vpc_id: vpc-0mine01 }
    cluster: { type: eks, name: mine-prod-euw1, addons: [core_dns, ebs_csi] }
    databases: [{ engine: postgres, version: "16.2", name: mine-db, observability: { mode: dbo11y } }]
workloads:
  - { type: web_service, name: mine-api, runs_on: mine-prod-euw1,
      traffic: { off_peak_rps: 5, peak_rps: 40 },
      endpoints: [{ route: "GET /v1/ping", error_rate: 0.01, p95_ms: 80 }] }

Incident scenarios

Declare named, reusable failure bundles in a blueprint, then fire them on a schedule or live via the control plane. Each effect names a mode, an optional target (instance name, <axis>:* wildcard, or omitted for a single-axis mode), and an intensity in [0,1].

scenarios:
  - name: db-pressure
    title: "Database under load"
    summary: "Connection saturation + slow queries hitting the production DB"
    effects:
      - { mode: connection_saturation, target: mine-db, intensity: 0.7 }
      - { mode: slow_query_storm,      target: mine-db, intensity: 0.5 }

incidents:
  # Schedule the whole scenario:
  - { scenario: db-pressure, at: "2026-06-19T14:00", for: 30m }
  # Or fire a single mode directly:
  - { kind: oom_kill, target: mine-prod-euw1, at: "2026-06-20T10:00", for: 10m, intensity: 0.6 }

Scenarios can also be activated or deactivated live without a restart — see Control plane below.

Control plane

The control plane (GET /control/ui, GET /control/schema, GET /control/state) is available by default. Mutation routes require CONTROL_TOKEN when set.

For the mutation examples below, run this once in Bash or Zsh. When CONTROL_TOKEN is set it writes credentials to a mode-0600 temporary netrc file, keeping the token out of process arguments. When the token is unset, requests remain unauthenticated:

control_auth=()
control_netrc=
if [ -n "${CONTROL_TOKEN:-}" ]; then
  control_netrc="$(mktemp)"
  chmod 600 "$control_netrc"
  printf 'machine localhost login control password %s\n' "$CONTROL_TOKEN" > "$control_netrc"
  control_auth=(--netrc-file "$control_netrc")
  trap 'test -z "$control_netrc" || rm -f "$control_netrc"' EXIT
fi

Route	Method	Description
/control/schema	GET	Blueprint-derived schema: all modes, addressable targets, scenarios, live scaling state.
/control/state	GET	Current control snapshot (volume multiplier, active scenarios, failures, scaling).
/control/scenarios	POST	Replace the active scenario list using qualified blueprint/scenario IDs. Prefer the item routes for ordinary activation/deactivation.
/control/scaling	POST	Set live workload pod counts using qualified blueprint/workload IDs, within blueprint-declared bounds. Node count cascades automatically.
/control/failures	POST	Ad-hoc {mode → {enabled, intensity, scope}} override (escape hatch; unknown modes warned, not rejected).
/control/load	POST	Master volume multiplier: scales the shape-driven synthetic volume; 0 pauses web_service, app and ai_agent traffic (see docs/control-plane.md).

Example — activate a scenario:

curl -fsS "${control_auth[@]}" -X POST http://localhost:8088/control/scenarios/activate \
  -H "Content-Type: application/json" \
  -d '{"scenario": "mine/db-pressure"}'

Example — scale workload pods live:

curl -fsS "${control_auth[@]}" -X POST http://localhost:8088/control/scaling \
  -H "Content-Type: application/json" \
  -d '{"mine/mine-api": 8}'
# Node count cascades: k8scluster + ec2 both re-derive via fixture.DeriveNodes.
# Scale-down retires old pod/node series automatically (state.DropWhere).

Capabilities and support

See the construct catalogue, workloads and signal contracts for supported telemetry. Vendor-specific gaps are recorded in cantfind.md; implementation work is tracked in Backlog. A modelled topology does not imply that every vendor telemetry profile is available.

The verification runbook explains how to prove delivery in your own stack. Passing a local dry run or a historical deployment test does not establish current live delivery.

License

synthkit is licensed under the GNU Affero General Public License v3.0 only (AGPL-3.0-only). See LICENSE and LICENSING.md. Every Go source file carries an SPDX-License-Identifier: AGPL-3.0-only header.