tailscale2otel - Tailscale → OpenTelemetry & Prometheus exporter

tailscale2otel exports Tailscale fleet state, network flows and configuration changes as OpenTelemetry metrics and logs over OTLP, through a Prometheus /metrics endpoint, or both. It runs as one static Go binary and supports a reduced collector set for Headscale. Optional traces describe the exporter’s own work.

Documentation · Getting started · Installation · Configuration · Metrics catalog

336 metrics + 31 log-event types	across 17 collectors
19 Tailscale API endpoints consumed	polled, streamed, or webhook-driven
135 shipped rules	112 alert + 23 recording, Grafana-managed
2 Grafana dashboards	tailnet + exporter health, v2 dynamic (Grafana 13+)
OTLP push (gRPC/HTTP)	and/or a Prometheus pull endpoint

Why this exists

Tailscale exposes device inventory, flow and audit logs, users, keys, DNS, ACL policy and device posture through its APIs. tailscale2otel combines those sources in your existing telemetry backend and derives expiry, traffic, configuration and collector-health signals. Cardinality controls bound the number of flow metric series.

What it adds

• Network flow logs as both metrics and logs. Low-cardinality aggregate counters (tailscale.network.io / .packets / .flows) for dashboards and alerting, plus full-fidelity per-connection records as OTEL logs for drill-down - with a top-N rollup (busiest 500 pairs, rest folded to __other__), opt-in port dimensions, and IANA service-name attribution so dst.port: 443 becomes https. Without the rollup, each endpoint pair becomes another metric series and the cost grows with the traffic graph.

• Configuration audit logs → structured OTEL logs + a curated, security-categorized change counter, so you can alert on high-value tailnet changes without ingesting the whole stream.

• Central tailscaled node-metrics polling. Scrapes each node's native client-metrics endpoint (:5252) from one place instead of deploying a scraper per node - with automatic target discovery from the devices API (tag include/exclude, online-only, address family). Emits both the raw tailscaled_* series and 8 curated tailscale.node.* metrics with folded low-cardinality attributes.

• Beyond the device API. Users, auth keys / OAuth clients / API tokens (with expiry), tailnet settings, DNS, ACL policy (scored for structural risk: wildcards, unrestricted rules, auto-approvers, SSH wildcards), device posture / MDM integrations, Tailscale Services, webhook endpoints, contacts, log-stream delivery health, and OAuth apps.

• Four ingestion paths into one pipeline - poll the API, receive Tailscale's log stream on a built-in Splunk-HEC-compatible receiver, read Tailscale's flow-log export straight out of an S3-compatible bucket, or take real-time HMAC-verified webhooks. Flow and audit sources share their respective processors; webhooks have their own event processor.

• Offline GeoIP and ASN enrichment of external peers. Optional, from MaxMind .mmdb files on local disk - no hosted lookup service, no per-address network call on the hot path. Country and continent are bounded and can go on flow metrics; the autonomous system (and, with a City database, locality and coordinates) ride the flow logs, without adding metric labels. Databases hot-swap on a schedule, with a built-in MaxMind updater if you want one. Tailnet addresses are never geolocated.

• Multi-tailnet / MSP mode - one process observing N tailnets, each with its own credentials, and tailscale.tailnet as a real label on every signal (no join required).

Start with a destination

• Grafana Cloud over OTLP pushes OpenTelemetry data to Grafana Cloud.
• Prometheus pull exposes /metrics for a scraper you already operate.
• stdout prints telemetry locally, with no observability backend.

Ready-made configuration starters live under examples/config/: Grafana Cloud OTLP, Prometheus pull, stdout, Headscale, and multi-tailnet/MSP. They keep credentials empty and document the environment or mounted-file forms to fill in.

The canonical runnable Docker, Compose, Helm, and binary commands live in Getting Started. It explains the terms, how to supply Tailscale authentication, and what result proves each path works. For installation details such as release binaries, persistence, and secret mounts, see Installation.

Where the telemetry goes

• OTLP push (otlp.protocol: grpc|http) with Grafana Cloud authentication: set otlp.grafana_cloud.{instance_id,token} and the Basic-auth header is built for you. Full TLS/mTLS knobs. Metrics and logs always; traces are opt-in (tracing.enabled) for the exporter's own self-observability, with exemplars linking API-duration histograms to the originating span.
• Prometheus pull endpoint (delivery.mode: prometheus or prometheus.enabled) - GET /metrics on its own loopback-only listener by default (127.0.0.1:2112), with bearer/basic auth and TLS available for remote scrapers. Use it if you already run Prometheus and don't want an OTLP pipeline.
• stdout for local debugging.

OTLP→Prometheus naming: query the normalized name. Dots become underscores, monotonic counters gain _total, units suffix (By→_bytes, s→_seconds), and a unit-1 gauge gains _ratio - so tailscale.network.io → tailscale_network_io_bytes_total. The full mapping is in the metrics catalog.

Collectors

Collector	Cadence	Emits
devices	60s	online/last-seen/key-expiry/update gauges, NAT & connectivity quality, per-DERP latency, subnet routes, tailnet lock, fleet hygiene roll-ups. Feeds the enrichment cache
flowlogs	60s	aggregated traffic counters + per-connection flow logs
auditlogs	60s	audit-event logs + a categorized change counter
k8s_audit	60s	(opt-in) tsrecorder Kubernetes API requests and terminal sessions from object storage
users	300s	user/role/status counts, per-user device & connection gauges, outstanding invites
keys	300s	expiry gauges and counts across auth keys, OAuth clients and API tokens
oauth_apps	300s	OAuth-application inventory (alpha API; idles silently where unavailable)
settings	600s	tailnet feature-toggle gauges
acl	600s	ACL size, change detection (by ETag), structural risk scoring
dns	600s	nameserver / search-path / split-zone counts, MagicDNS flag
contacts	600s	contact verification status (the email itself is never emitted)
webhooks	600s	webhook-endpoint inventory + per-endpoint subscription counts
posture_integrations	600s	MDM/EDR integration counts, sync health, matched devices
log_stream	600s	Tailscale's own SIEM-sink delivery health + delivery counters
services	600s	Tailscale Services (VIP) inventory - counts, ports, opt-in backing hosts
pam	600s inventory / 60s sessions	(opt-in) Border0 PAM inventory, configuration and session telemetry; see PAM

Each can be disabled or re-tuned. Under provider: headscale the Tailscale-only collectors auto-disable and a reduced set (devices, users, keys, ACL, node-metrics) runs.

Device enrichment depends on the devices collector - flow/audit IP→name resolution silently degrades to unknown/external without it.

Logs: poll, stream or object store - pick one

Both flowlogs and auditlogs take a source of poll (default), stream, objectstore or both. Tailscale exports each log type to object storage independently, so each has its OWN destination (collectors.flowlogs.objectstore / collectors.auditlogs.objectstore) and nothing is inherited between them. Pick exactly one method per log type - running two risks double-counting, cross-source de-dup is only a best-effort failsafe, and the exporter WARNs at startup when it sees this.

# Poll: tailscale2otel pulls on a schedule (interval/lag/initial_lookback/max_window apply).
flowlogs: { enabled: true, source: poll, interval: 60s, lag: 120s, initial_lookback: 5m, max_window: 1h }

# Stream: Tailscale pushes to the built-in HEC receiver (the window fields are ignored).
flowlogs: { enabled: true, source: stream, log_mode: per_connection }

# Object store: read the export Tailscale writes to S3. No API quota, and the cheapest path for a
# busy tailnet. Credentials come from the ambient chain (env, IRSA, instance profile).
flowlogs:
  enabled: true
  source: objectstore
  objectstore: { endpoint: https://s3.eu-west-2.amazonaws.com, region: eu-west-2, bucket: my-flow-logs }

Object-store delivery is at-least-once. With the file checkpoint store, successful object identities and failed-object gaps survive restart; transient failures retry with bounded backoff, while invalid compressed objects are quarantined for operator acknowledgement. One object is all-or-nothing - every row is decoded before any is committed, so a mid-object failure emits nothing rather than a partial prefix - but the object as a unit replays if the process dies between emission and the checkpoint write. OTLP/backend acknowledgement is outside this boundary.

Backfill has a hard ceiling of 14 day partitions - today plus the previous 13 days - under the default layout: partitioned, whatever initial_lookback says. It is permanent, not per-cycle: one cycle enumerates at most 14 day prefixes newest-first, and the cursor only moves forward, so older days are never listed and are skipped with no gap, no error and no metric. layout: flat has no partitions to cap and reaches arbitrarily far back, at the cost of more LIST requests. The exporter warns at startup when initial_lookback exceeds the ceiling. See Streaming & webhooks for the full path-by-path compatibility, delivery and durability matrix.

Checkpoints persist how far poll and object-store collectors have read. Details on all paths, receiver auth, object-gap handling, and auto_configure are in Streaming & webhooks.

Dashboards, alerts & the admin UI

• Dashboards - deploy/grafana/ ships 2 dashboards on Grafana's v2 schema (Grafana 13+): Tailnet (is my tailnet healthy - devices, network, security, policy) and Exporter health (is the exporter healthy - collection, ingestion, delivery, runtime, cost), cross-linked to each other, with dynamic rendering so a section only appears when its data is present. Grafana 13+ is a hard requirement - 12.4 accepts the file with a 200 and renders nothing, and 11.5 rejects it with the misleading Dashboard title cannot be empty. This repository publishes them through GitSync; other deployments can import or provision the two JSON resources. See Dashboards.
• Alerts - deploy/alerts/grafana-managed/ ships 112 alert rules and 23 recording rules (135 total) as rules.alerting.grafana.app manifests, one JSON per rule. Push them with gcx resources push -p deploy/alerts/grafana-managed. Every alert carries a runbook_url, and 108 of 112 link a canonical dashboard panel. See Alerts and Runbooks.
• Admin status page - on by default at 127.0.0.1:9091. Liveness/readiness probes at /healthz and /readyz (never auth-gated), a live status page at /, and the same snapshot at /api/status.json: per-collector health, active-series cardinality with per-label breakdown, the full metrics/log catalog, discovered node targets, and a redacted config summary. Entirely self-contained - no CDN assets, so it renders on an air-gapped tailnet. Auth fails closed on a non-loopback bind with no admin.auth.token.
• Local explorers provide a flow view, an , JSON/CSV flow exports, and a . They use the admin authentication boundary.

Configuration

Layered, lowest precedence first: built-in defaults → optional YAML file → environment variables. Scalar and simple-list fields are settable as TS2OTEL_ + the dotted key path with __ between levels; maps and lists of structured entries stay in YAML:

Config key	Environment variable
tailscale.auth.oauth.client_secret	TS2OTEL_TAILSCALE__AUTH__OAUTH__CLIENT_SECRET
otlp.endpoint	TS2OTEL_OTLP__ENDPOINT
collectors.flowlogs.interval	TS2OTEL_COLLECTORS__FLOWLOGS__INTERVAL

An unrecognised TS2OTEL_* variable is logged as a WARN at startup - usually a typo.

Authentication: prefer an OAuth client (auto-refreshing, least-privilege all:read) over an API key. Keyless workload identity (OIDC token exchange, e.g. a Kubernetes projected service-account token) is also supported, and supported file-backed credentials have a *_file variant for Docker/Kubernetes secrets. PAM uses TS2OTEL_PAM__TOKEN and has no token_file setting.

OAuth access tokens refresh automatically. Selected TLS and credential files can rotate in place; other configuration changes require a restart. See the reload reference.

→ Full configuration reference · every TS2OTEL_* variable · config.example.yaml

Documentation

Getting started	First telemetry through OTLP, Prometheus or stdout
Installation	Docker, Helm, compose, binaries
Configuration	Every key, default and gotcha
Metrics catalog	All 336 metrics and 31 log events
Node metrics	Central tailscaled scraping
Streaming & webhooks	HEC receiver and webhooks
Architecture	How it fits together
Security	Data handling, PII, receiver auth
Troubleshooting	When it doesn't work
Runbooks	Alert investigation and remediation

Development

just setup
just check

just load-wal runs the local WAL throughput comparison with synthetic traffic and a loopback OTLP sink; it does not use a live tailnet.

Small single-purpose packages under internal/: telemetry (OTEL facade), collector (scheduler/registry/checkpoints + one package per source), tsapi (Tailscale client), provider/hsapi (control-plane abstraction + Headscale), flowlog/audit (records + processors), enrich (device cache), rdns, config, and the stream/webhook receivers. Generated artifacts have family-specific gen-* recipes. Run just gen or the affected family when changing their inputs. just --list shows the supported tasks; just check covers the root and four tool modules.

API drift CI

Tailscale's API and OpenAPI spec evolve continuously ("may change or break without notice"), which has broken decoders here before. The repository runs these checks:

Lane	When	What it checks
Schema-driven decode tests	every PR (gates)	synthesizes payloads from the vendored OpenAPI spec + known wire quirks (numeric proto, polymorphic audit old/new) through the real decoders, plus a boundary matrix running every consumed operation against every boundary shape - null, empty container, nulled nullable fields, extreme values, an unknown enum member, an additive field, and a wrong container shape that must be rejected. Runs inside the normal go test -race ./... leg, which ci-success requires
Exploratory fuzzing	pushes to main (advisory)	go test -fuzz over the HEC envelope, HEC timestamps and the flow/audit decoders. Deliberately not required: finding a NEW crasher is nondeterministic, so gating it would let an unrelated PR randomly block merges. Each target's seed corpus runs in the gated leg above, so a KNOWN crasher still blocks
OpenAPI drift	daily	diffs the live spec against the vendored copy, scoped to consumed operations. Covers response fields, path/query/header parameters (requiredness, type, default, enum), the success-status set and request/response media types, classifying each as breaking, behavioral or additive
Client-lib tracking	weekly	builds and tests against tailscale-client-go/v2@main and @latest
Scheduled fuzzing	weekly	the same nine fuzz targets for 15 minutes each instead of 120 seconds, where a nondeterministic finding costs nobody a blocked merge. Opens a deduplicated tracking issue on a crasher and attaches the failing input
Live contract	daily	hits the real API read-only and asserts every consumed GET still decodes
Changelog review	monthly	reads Tailscale's changelog feed for entries that name something this exporter collects and carry no recorded verdict in spec/changelog-reviewed.json. Catches a capability announced before, or without, any OpenAPI change. Reviewing an entry means recording a verdict - including a negative one, so a surface already declined is never re-proposed

Scheduled lanes are advisory - they open a deduplicated tracking issue and fail the run, but never block PRs. Exploratory fuzzing is advisory too, for the reason in its row, so it runs on pushes to main rather than on pull requests; of the PR-time lanes, the schema-driven decode tests gate. The seed corpora ride go test -race, so a known crasher still blocks a merge.

The cadences and the advisory-versus-gating split in this table are asserted by internal/ci/workflowcontract_test.go, which reads the workflow files - two of these rows claimed "weekly" against daily crons until that test was added (#436).

gh label create api-drift -c FBCA04
gh label create clientlib-drift -c FBCA04
gh label create live-contract -c FBCA04

The live lane stores no long-lived Tailscale key. It runs on a standard GitHub-hosted runner and mints a short-lived token from Tailscale's OAuth endpoint using a read-only (all:read) OAuth client, whose TS_OAUTH_CLIENT_ID / TS_OAUTH_CLIENT_SECRET are repo secrets. Keeping them as secrets is safe because this lane is schedule + workflow_dispatch only, so a fork PR can never run it and never reach them; the minted token is masked and lives only for that run. Set the repo variable TS_TAILNET (the tailnet name is not a secret). Missing configuration fails the lane loudly rather than self-skipping, so a misconfigured preflight cannot look green. Optionally set the ANTHROPIC_API_KEY secret for Claude enrichment on the spec-drift and live lanes; the client-lib lane never receives it by design, since it builds untrusted upstream code.

License

Apache License 2.0 - full text in LICENSE; third-party attribution and bundled notices/SBOMs in LICENSING.md.