Prometheus Proxy

Prometheus is an excellent systems monitoring and alerting toolkit, which uses a pull model for collecting metrics. The pull model is problematic when a firewall separates a Prometheus server and its metrics endpoints.

Prometheus Proxy enables Prometheus to scrape metrics endpoints running behind a firewall and preserves the native pull-based model architecture.

Table of Contents

• New Features
• Architecture
• Quick Start
• Homebrew
• Building from Source
• Configuration Examples
• Docker Usage
• Advanced Features
• Monitoring & Observability
• Configuration Options
• Security & TLS
• Troubleshooting
• Documentation
• License

✨ New Features

Version 4.4.0 (released 2026-10-29) is a security and reliability release. An authenticated agent can no longer complete another agent's scrape or stall the proxy with a long path, the proxy bounds what agents report about themselves, and builds that embed the agent now get the patched Netty. Service discovery gives Prometheus targets it accepts, now also behind HTTPS or a path prefix; the Helm charts can take their config from a Secret (existingSecret); and the agent passes the scrape timeout on to exporters, sends a target URL's credentials with the first request, and reads no more of a response into memory than maxContentLengthMBytes. Some agent configs that start today will fail at startup, and the proxy chart's readiness probe moves to /ping, so read Before you upgrade in the release notes first; the changelog has the full list.

Version 4.3.0 (released 2026-10-12) adds three opt-in JVM metric groups (buffer pools, JIT compilation, and native memory), shows each agent's real address on the dashboard instead of Unknown, and makes a proxy that fails to start exit instead of hanging. The images move to the Ubuntu 24.04 variant of their eclipse-temurin base, which is clear of an OpenSSL vulnerability, and gRPC, netty-tcnative, and Kotlin are updated; the changelog has the full list.

Version 4.2.0 (released 2026-09-26) moves the proxy and agent to the Prometheus Java client 1.x, replacing the unmaintained 0.x client. Every proxy_* and agent_* metric keeps its name, labels, and buckets, but the _created series are gone and, with the JVM exports on, the JVM memory metrics are renamed, so read Before you upgrade in the release notes first. The proxy and agent can now be installed with Homebrew, the bundled Grafana dashboards' connection, eviction, heartbeat, and scrape-count panels show data again, and the JARs no longer print logback and kotlin-logging lines at startup. It also adds Helm charts for the proxy and the agent, and ships the alerting rules as grafana/alerts.yml; the changelog has the full list.

Version 4.1.0 (released 2026-09-23) is a security and reliability release: agents can no longer answer, read, or take over each other's scrapes and paths, a rejected path no longer takes an agent offline and a rejecting proxy is now failed over, and the proxy bounds what clients and agents can ask of it (scrape backlogs, in-flight scrapes, paths per agent, redirects, and service-discovery labels). It changes some defaults, scrape outcome labels, and startup checks, so read Before you upgrade in the release notes first; the changelog has the full list.

Version 4.0.1 (released 2026-07-31) is a dashboard patch release: agents are now identified by name rather than internal id in the path layout, 29 WCAG AA contrast failures are fixed (measured floor 4.59:1), and the page is usable with a screen reader — see the changelog for the full list. Nothing in it changes configuration, metrics, or the wire protocol.

Version 4.0.0 (released 2026-07-25) added five features focused on running the proxy as production infrastructure:

• Operational Dashboard — A read-only, live-updating web page that answers "why isn't this target scraping?" without log-diving on two machines. It shows every connected agent, every registered path — including paths whose agent has gone, precisely the case that used to vanish from view — and how recent scrapes went.
• Proxy High Availability — The proxy is no longer a single point of failure for the entire metrics plane. Agents take an ordered list of proxy endpoints, fail over to a standby when the primary is unreachable, and return to the primary when it recovers — no health prober, no manual step.
• Per-Agent Identities and Path Authorization — A shared token can't tell agents apart, so any holder could register — and silently take over — any path. Named identities scope each agent to its own path patterns, enable per-agent revocation, and make one proxy safe to share across teams. Existing shared tokens keep working during migration.
• Metric Filtering at the Agent — Drop unwanted metric families before they cross the WAN, the one place where dropping data pays the most and where Prometheus's own metric_relabel_configs can't reach. Per-path allow/deny regexes with Prometheus-compatible anchoring; whole families are kept or dropped atomically.
• Dynamic Target Discovery — Add and remove scrape targets by editing a watched file — no agent restart, and unchanged paths never stop scraping. The file can be generated (Kubernetes ConfigMap, Ansible, cron), so automation only needs permission to write a file, never to restart a process.

🏗️ Architecture

📖 Docs site: Architecture for the full component breakdown and request flow, and the Glossary for the core terms used throughout the docs.

The prometheus-proxy runtime comprises two services:

• proxy: runs in the same network domain as Prometheus server (outside the firewall) and proxies calls from Prometheus to the agent behind the firewall.
• agent: runs in the same network domain as all the monitored hosts/services/apps (inside the firewall). It maps the scraping queries coming from the proxy to the actual /metrics scraping endpoints of the hosts/services/apps.

Prometheus Proxy solves the firewall problem by using a persistent gRPC connection initiated from inside the firewall. Here's a simplified network diagram of how the deployed proxy and agent work:

Endpoints running behind a firewall require a prometheus-agent (the agent) to be run inside the firewall. An agent can run as a stand-alone server, embedded in another java server, or as a java agent. Agents connect to a prometheus-proxy (the proxy) and register the paths for which they will provide data. One proxy can work with one or many agents.

Components

• 🌐 Proxy - Runs outside the firewall alongside Prometheus

  • HTTP server (port 8080) - Serves metrics to Prometheus
  • gRPC server (port 50051) - Accepts agent connections
  • Service discovery support for dynamic targets

• 🔗 Agent - Runs inside the firewall with monitored services

  • Connects to proxy via gRPC (outbound connection only)
  • Scrapes local metrics endpoints
  • Registers available paths with proxy

Key Benefits

• ✅ Firewall-friendly - Only requires outbound connection from agent
• ✅ Preserves pull model - Prometheus continues to pull as normal
• ✅ High performance - Built with Kotlin coroutines and gRPC
• ✅ Secure - Optional TLS with mutual authentication, plus per-agent identities with path authorization
• ✅ Scalable - One proxy supports many agents
• ✅ Highly available - Agents fail over across an ordered list of proxies, and back again
• ✅ Dynamic - Agents pick up target changes from a watched file, no restart needed
• ✅ Bandwidth-conscious - Optional per-path metric filtering drops unwanted families at the agent
• ✅ Observable - A read-only live dashboard shows every agent, path, and recent scrape in one place
• ✅ Zero changes to existing Prometheus configuration patterns

🚀 Quick Start

📖 Docs site: Quick Start guide walks through these steps with more detail.

Requirements: Java 17 or newer

CLI Quick Start

1. Download the latest proxy and agent JAR files from releases

2. Start the proxy (runs outside the firewall with Prometheus):

   java -jar prometheus-proxy.jar
   

3. Start the agent (runs inside the firewall with your services):

   java -jar prometheus-agent.jar \
     -Dagent.proxy.hostname=mymachine.local \
     --config https://raw.githubusercontent.com/pambrose/prometheus-proxy/master/examples/myapps.conf
   
   # or use --proxy option
   java -jar prometheus-agent.jar \
     --proxy mymachine.local \
     --config https://raw.githubusercontent.com/pambrose/prometheus-proxy/master/examples/myapps.conf
   

4. Configure Prometheus to scrape from the proxy at http://mymachine.local:8080

5. Verify it works with:

   curl -s http://mymachine.local:8080/app1_metrics | head
   
   # or, if SD is enabled
   curl -s http://mymachine.local:8080/discovery | jq '.'
   

6. Steps 2 and 3 run in the foreground until Ctrl+C. To keep the proxy and agent running after you close the terminal, start each with nohup, sending its output to a log file and saving its process ID:

   # On the proxy's machine
   nohup java -jar prometheus-proxy.jar > proxy.log 2>&1 &
   echo $! > proxy.pid
   
   # On the agent's machine
   nohup java -jar prometheus-agent.jar --proxy mymachine.local --config myapps.conf > agent.log 2>&1 &
   echo $! > agent.pid
   
   # Later, to follow a log or stop a process
   tail -f proxy.log
   kill "$(cat proxy.pid)"
   

   Nothing restarts them if they exit or the machine reboots; brew services and Docker's --restart, below, do.

🍺 Homebrew

On macOS and Linux, the proxy and the agent are also available from Homebrew. Each formula installs the Java it runs on (openjdk@25), so there is no JAR to download:

# On the proxy's machine, outside the firewall
brew install pambrose/tap/prometheus-proxy
prometheus-proxy

# On the agent's machine, inside the firewall
brew install pambrose/tap/prometheus-agent
prometheus-agent --proxy mymachine.local --config myapps.conf

To keep either one running in the background, edit its config in $(brew --prefix)/etc/ and start it with brew services, which also starts it again at every login: prometheus-proxy.conf sets the proxy's ports and agent authentication, and prometheus-agent.conf sets the proxy the agent connects to and the paths it serves. Each logs to $(brew --prefix)/var/log/<name>.log.

brew services start prometheus-proxy     # start now and at every login
brew services start prometheus-agent

brew services restart prometheus-agent   # pick up an edited config
brew services stop prometheus-agent      # stop, and no longer start at login

🛠️ Building from Source

If you prefer to build the project from source:

1. Clone the repository:

   git clone https://github.com/pambrose/prometheus-proxy.git
   cd prometheus-proxy
   

2. Build the fat JARs:

   ./gradlew agentJar proxyJar
   

   agentJar and proxyJar are dedicated ShadowJar tasks; the default shadowJar task is disabled so it cannot produce a third redundant fat jar.

3. The JARs will be available in build/libs/:

• build/libs/prometheus-proxy.jar
• build/libs/prometheus-agent.jar

Docker Quick Start

# Start proxy
docker run --rm -p 8080:8080 -p 50051:50051 pambrose/prometheus-proxy:4.4.0

# Start agent
docker run --rm \
  --env AGENT_CONFIG='https://raw.githubusercontent.com/pambrose/prometheus-proxy/master/examples/simple.conf' \
  pambrose/prometheus-agent:4.4.0

These run in the foreground until Ctrl+C. To run a container in the background instead, and have it start again if it exits or Docker restarts, replace --rm with --detach --name <name> --restart unless-stopped (Docker doesn't allow --rm with --restart):

docker run --detach --name prometheus-proxy --restart unless-stopped \
  -p 8080:8080 -p 50051:50051 pambrose/prometheus-proxy:4.4.0

docker logs --follow prometheus-proxy   # follow its log
docker stop prometheus-proxy            # stop it, and no longer restart it
docker rm prometheus-proxy              # remove the stopped container

📋 Configuration Examples

📖 Docs site: Agent configuration and Proxy configuration document every config key in depth.

Agent Configuration

If the prometheus-proxy were running on a machine named mymachine.local and the agent.pathConfigs value in the myapps.conf config file had the contents:

agent {
  pathConfigs: [
    {
      name: "App1 metrics"
      path: app1_metrics
      labels: "{\"key1\": \"value1\", \"key2\": 2}"
      url: "http://app1.local:9100/metrics"
    },
    {
      name: "App2 metrics"
      path: app2_metrics
      labels: "{\"key3\": \"value3\", \"key4\": 4}"
      url: "http://app2.local:9100/metrics"
    },
    {
      name: "App3 metrics"
      path: app3_metrics
      labels: "{\"key5\": \"value5\", \"key6\": 6}"
      url: "http://app3.local:9100/metrics"
    }
  ]
}

then the prometheus.yml scrape_config would target the three apps with:

• http://mymachine.local:8080/app1_metrics
• http://mymachine.local:8080/app2_metrics
• http://mymachine.local:8080/app3_metrics

If the endpoints were restricted with basic auth/bearer authentication, you could either include the basic-auth credentials in the URL with: http://user:pass@hostname/metrics or they could be configured with basic_auth/ bearer_token in the scrape-config. The agent sends credentials from the URL with each request, unless the scrape-config supplies an Authorization header, which is sent instead.

Corresponding Prometheus Configuration

The prometheus.yml file would include:

scrape_configs:
  - job_name: 'app1 metrics'
    metrics_path: '/app1_metrics'
    bearer_token: 'eyJ....hH9rloA'
    static_configs:
      - targets: [ 'mymachine.local:8080' ]
  - job_name: 'app2 metrics'
    metrics_path: '/app2_metrics'
    basic_auth:
      username: 'user'
      password: 's3cr3t'
    static_configs:
      - targets: [ 'mymachine.local:8080' ]
  - job_name: 'app3 metrics'
    metrics_path: '/app3_metrics'
    static_configs:
      - targets: [ 'mymachine.local:8080' ]

🐳 Docker Usage

📖 Docs site: Docker deployment guide covers compose files, bind mounts, and container configuration. Running on Kubernetes? See the Kubernetes guide.

Multi-Platform Images

The docker images support multiple architectures (amd64, arm64, s390x, ppc64le):

docker pull pambrose/prometheus-proxy:4.4.0
docker pull pambrose/prometheus-agent:4.4.0

Production Docker Setup

Start a proxy container with:

# Proxy with admin and metrics enabled
docker run --rm -p 8082:8082 -p 8092:8092 -p 50051:50051 -p 8080:8080 \
        --env ADMIN_ENABLED=true \
        --env METRICS_ENABLED=true \
        --restart unless-stopped \
        pambrose/prometheus-proxy:4.4.0

Start an agent container with:

# Agent with remote config file
docker run --rm -p 8083:8083 -p 8093:8093 \
        --env AGENT_CONFIG='https://raw.githubusercontent.com/pambrose/prometheus-proxy/master/examples/simple.conf' \
        --restart unless-stopped \
        pambrose/prometheus-agent:4.4.0

Or use Docker Compose: etc/compose/proxy.yml runs a proxy, an agent, and a Prometheus server that scrapes through them (docker compose -f etc/compose/proxy.yml up).

Using the config file simple.conf, the proxy and the agent metrics would be available from the proxy on localhost at:

• http://localhost:8082/proxy_metrics
• http://localhost:8083/agent_metrics

If you want to use a local config file with a docker container (instead of the above HTTP-served config file), use the docker mount option. Assuming the config file prom-agent.conf is in your current directory, run an agent container with:

# Agent with local config file
docker run --rm -p 8083:8083 -p 8093:8093 \
    --mount type=bind,source="$(pwd)"/prom-agent.conf,target=/app/prom-agent.conf \
    --env AGENT_CONFIG=prom-agent.conf \
    pambrose/prometheus-agent:4.4.0

Note: The WORKDIR of the proxy and agent images is /app, so make sure to use /app as the base directory in the target for --mount options.

⚙️ Advanced Features

Embedded Agent

If you are running a JVM-based program, you can run with the agent embedded directly in your app and not use an external agent. This approach eliminates the need for a separate agent process when your application already runs on the JVM.

📖 Docs site: Embedded Agent guide for the full API surface and lifecycle details.

// Start embedded agent; false (exitOnMissingConfig) makes config errors throw instead of exiting the JVM
EmbeddedAgentInfo agentInfo = startAsyncAgent("configFile.conf", false);

// Your application code runs here
// The agent runs in the background and does not block your application

// Shutdown the agent when the application terminates
agentInfo.shutdown();

If the configuration cannot be loaded (a parse error, an unreachable config URL, or a missing file), the embedded startAsyncAgent / startSyncAgent entry points throw io.prometheus.common.ConfigLoadException for your application to catch, rather than terminating the host JVM via exitProcess. Stand-alone agents (run from the CLI) still exit on a missing or unreadable config. If the agent then fails to start (for example, its admin or metrics port is already in use), startAsyncAgent throws IllegalStateException with the startup failure as its cause, after releasing everything the agent had opened.

Several agents can run in one JVM with metrics on, at once or one after another: they share one set of agent metrics, and each series carries its agent's launch_id. An embedded agent's logLevel sets only the io.prometheus loggers, so it leaves your application's own logging alone.

Service Discovery

Enable Prometheus service discovery support:

java -jar prometheus-proxy.jar \
  --sd_enabled \
  --sd_path discovery \
  --sd_target_prefix proxy-host:8080

Access discovery endpoint at: http://proxy-host:8080/discovery

--sd_target_prefix is the proxy's address as Prometheus reaches it: host:port (the default is localhost:8080), or an http or https URL such as https://proxy.example.com/prometheus/, whose scheme and path the proxy gives each target as its __scheme__ label and a prefix of its __metrics_path__.

📖 Docs site: Service Discovery guide explains the discovery payload format and the Prometheus http_sd_configs wiring.

Dynamic Target Discovery (Agent)

By default the agent's scrape targets come from the static agent.pathConfigs list, read once at startup — adding or removing a target means editing the config and restarting the agent. Enable dynamic discovery to have the agent reconcile its registered paths against a watched file at runtime, with no restart:

agent {
  discovery {
    enabled = true
    file.path = "/etc/prometheus-proxy/targets.conf"   // HOCON or JSON list of paths
    reconcileIntervalSecs = 30                          // Poll and full-resync interval
  }
}

The discovery file is a paths list of the same { name, path, url, labels } entries as pathConfigs:

paths = [
  { name = "app1", path = "app1_metrics", url = "http://app1:9090/metrics" }
  { name = "app2", path = "app2_metrics", url = "http://app2:9090/metrics" }
]

Every reconcileIntervalSecs the agent reads the file and registers new paths, unregisters ones no longer listed, and re-registers a path whose URL or labels changed — without disturbing the paths that did not change. Behavior worth knowing:

• Additive to the static baseline. Discovered paths add to agent.pathConfigs (which is never touched); on a path collision the static entry wins. Leave pathConfigs empty to run discovery-only.
• Fail-safe reads. A missing/unreadable/malformed file keeps the last-known-good set; a valid-but-empty file removes all discovered paths.
• Polling, not inotify. Reliable under Kubernetes ConfigMap updates and bind mounts, where file watches often miss changes.
• Config-file only. discovery.file.path points at a list, so there is no CLI/env equivalent (though enabled, the path, and the interval are scalars you can also set via -D).

Because the file is a plain HOCON/JSON list, it can be generated — by a Kubernetes ConfigMap, an Ansible template, or a cron job querying a service registry — so your automation only needs permission to write a file, not to restart processes. A fully annotated example ships as examples/discovery-targets.conf.

This is distinct from the Prometheus Service Discovery above: that exposes a discovery endpoint so Prometheus can find proxied targets, whereas this lets the agent pick up target changes behind the firewall without a restart.

Metric Filtering (Agent)

By default the agent forwards every metric a target exposes, so chatty or high-cardinality endpoints pay full bandwidth across exactly the network boundary this product exists to bridge. An optional per-path filter drops whole metric families at the agent, before the payload is gzipped and chunked, cutting WAN bandwidth and downstream Prometheus cardinality at the source:

agent {
  pathConfigs: [
    { name: "app1", path: "app1_metrics", url: "http://app1:9090/metrics" }
  ]

  filters: [
    {
      path: "app1_metrics"          // The pathConfigs (or discovered) path this applies to
      metricNameAllow: []           // Fully-anchored regexes; empty allows every family
      metricNameDeny: [ "go_gc_.*", "go_memstats_.*" ]
    }
  ]
}

Both list fields are required on every element — write metricNameAllow: [] for a deny-only filter. The default is filters: [], so an existing config with no filters key loads unchanged. Behavior worth knowing:

• Regexes are fully anchored, matching Prometheus relabel_config semantics: "go_" matches nothing, "go_.*" is required to match go_goroutines. metricNameDeny is applied after metricNameAllow, so deny wins on overlap.
• Matching is per metric family, not per series. A # HELP/# TYPE line opens a family and the verdict is computed once, so a histogram's _bucket, _sum, and _count series are always kept or dropped together with their metadata lines — a filter can never deliver a histogram with some of its series missing.
• Fails open. A non-text Content-Type (protobuf, an HTML error body) or a body that is not valid UTF-8 passes through unfiltered and byte-exact. The worst case is bandwidth not saved, never a corrupted payload.
• Applies to discovered paths too. Filters are matched by path, so they cover both pathConfigs entries and paths added by Dynamic Target Discovery.
• Observable. agent_filter_lines_dropped_total and agent_filter_bytes_saved_total, labeled by launch_id and path, report what each filter is actually removing.
• Config-file only. filters is a list, so there is no CLI/env equivalent.

An invalid regex fails agent startup rather than surfacing later on a scrape. Not implemented: dropLabels, metric renaming/relabeling, and an agent-global filter — every filter is per-path.

Proxy Failover (High Availability)

A single proxy is a single point of failure for every target behind every agent — losing it (or just restarting it to change config) blinds Prometheus to everything on the far side of the firewall at once. For redundancy, give the agent an ordered list of proxy endpoints instead of one hostname:

agent {
  proxy {
    endpoints = [ "proxy-a.example.com:50051", "proxy-b.example.com:50051" ]
  }
}

Or comma-separated on the command line / environment (either replaces endpoints entirely):

java -jar prometheus-agent.jar --proxy proxy-a.example.com:50051,proxy-b.example.com:50051
# or: PROXY_HOSTNAME=proxy-a.example.com:50051,proxy-b.example.com:50051

The agent uses the first endpoint that accepts it. A failed connect moves it to the next, and so does a proxy that accepts the connection but rejects the agent's registration, or rejects every one of its static paths for a reason that can't clear. A connection that registered and later dropped sends it back to the head of the list, so a recovered primary is picked up on the next reconnect with no health prober and no manual step. Only one connection is active at a time, and a single value behaves exactly as before — existing configs need no change.

Two things to know before deploying a pair:

• Scrape both proxies with static_config, not http_sd_config. A standby proxy has no agents, so its service-discovery endpoint returns an empty list — which Prometheus treats as "these targets no longer exist" and silently deletes them: no up=0, no alert. With static_config the standby 404s, Prometheus records up=0, and your alerting works unchanged.
• All endpoints share one TLS configuration (one trust store, one overrideAuthority), so proxies with different CAs or certificate SANs fail with an opaque handshake error.

📖 Docs site: High availability for the full Prometheus-side setup, and Running an HA pair for how the dashboard shows failed-over agents.

Performance Tuning

Configure concurrent scraping:

java -jar prometheus-agent.jar \
  --max_concurrent_clients 5 \
  --client_timeout_secs 30 \
  --config myconfig.conf

📊 Monitoring & Observability

📖 Docs site: Monitoring guide for the full metrics reference and admin endpoints, plus Grafana & Alerting for ready-to-import dashboards and alert rules.

Operational Dashboard

The proxy ships a read-only, live-updating dashboard that answers the question this system raises most often — why isn't this target scraping? — without log-diving on two machines. It shows every connected agent, every registered path (including paths whose agent has gone, which would otherwise vanish exactly when you go looking), and how recent scrapes went:

java -jar prometheus-proxy.jar --dashboard    # or DASHBOARD_ENABLED=true / proxy.dashboard.enabled

Then open http://proxy-host:8094/dashboard. Two layouts share the page: Agents (drill into one agent) and Paths (one row per target). Both name the agent serving a path, so the two layouts can be read against each other. It runs on its own port — not the admin port — so it can be firewalled without cutting off Kubernetes health probes, and it loads no CDN assets, so it works in airgapped deployments. Like the admin and metrics endpoints it has no authentication or TLS: keep the port internal.

Keeping the port internal does not stop a web page from reaching the dashboard through the browser of someone who can. The dashboard refuses a WebSocket from a foreign site; behind a reverse proxy that rewrites the Host header, list the public origin in proxy.dashboard.allowedOrigins. It answers to any host name by default, which leaves it open to DNS rebinding (a page pointing its own DNS name at the dashboard's address); list the names you reach it by in proxy.dashboard.allowedHosts to close that. See Dashboard exposure.

Accessibility: text and UI colors clear the WCAG AA contrast thresholds with a measured floor of 4.59:1, the path table carries proper header semantics, and connection loss and recovery are announced to screen readers rather than signalled by colour alone.

📖 Docs site: Dashboard guide for both layouts, the status-bar gauges, and HA-pair behavior.

Built-in Metrics

Both proxy and agent expose their own metrics:

• Proxy metrics: http://proxy-host:8082/metrics
• Agent metrics: http://agent-host:8083/metrics. The agent sits behind the firewall, so give it a path for its own metrics and let Prometheus scrape them through the proxy, like any other target
• Admin endpoints: http://host:admin-port/ping, /healthcheck, /version
• JVM metrics: off by default; each group is turned on under proxy.metrics or agent.metrics in the config file (memoryPoolsExportsEnabled, garbageCollectorExportsEnabled, and so on). Version 4.3.0 adds buffer pools (bufferPoolExportsEnabled), JIT compilation (compilationExportsEnabled), and native memory (nativeMemoryExportsEnabled, which needs the JVM started with -XX:NativeMemoryTracking=summary); the Monitoring guide lists every metric

🔧 Configuration Options

📖 Docs site: Configuration overview and the CLI Reference document every option and environment variable.

The proxy and agent use the Typesafe Config library. Configuration values are evaluated in order: CLI options → environment variables → config file values.

Typesafe Config highlights include:

• support for files in three formats: Java properties, JSON, and a human-friendly JSON superset (HOCON)
• config files can be files or urls
• config values can come from CLI options, environment variables, Java system properties, and/or config files.
• config files can reference environment variables

📖 Complete configuration reference: CLI Options & Environment Variables Reference

Common Options Summary

Component	Option	Env Var	Description
Both	--config, -c	PROXY_CONFIG / AGENT_CONFIG	Path or URL to config file
Both	--admin, -r	ADMIN_ENABLED	Enable admin/health-check endpoints
Both	--metrics, -e	METRICS_ENABLED	Enable internal metrics collection
Both	--agent_token	AGENT_TOKEN	Pre-shared agent auth token; both sides must match (Default: disabled). For per-agent identities see proxy.auth
Proxy	--port, -p	PROXY_PORT	Port for Prometheus to scrape (Default: 8080)
Proxy	--agent_port, -a	AGENT_PORT	Port for Agents to connect via gRPC (Default: 50051)
Proxy	--dashboard	DASHBOARD_ENABLED	Enable the read-only operational dashboard (Default: disabled)
Proxy	--dashboard_port	DASHBOARD_PORT	Port for the operational dashboard (Default: 8094)
Proxy	--dashboard_host	DASHBOARD_HOST	Listen address for the dashboard (Default: 0.0.0.0)

Configuration Notes

• Formats: Supports HOCON (.conf), JSON (.json), and Java Properties (.properties).
• Logging: Customize with -Dlogback.configurationFile=/path/to/logback.xml. To pick up edits without a restart, set scan="true" on that file's <configuration> element, as logback/docker-logback.xml does; the config bundled in the JARs can't be watched.
• Dynamic Props: Use -Dproperty.name=value for any scalar configuration key (parsed as Java properties, so list/object values like proxy.auth must be set in a config file).
• Per-Agent Auth: proxy.auth (per-agent identities with path authorization) is a list of objects, so it is config-file-only — see Per-Agent Identities and Path Authorization.
• Keepalives: See the gRPC keepalive guide for tuning details.
• Exposition formats: Scrapes are carried as text, so the agent asks targets only for the text and OpenMetrics formats. Native histograms, which Prometheus receives only in the protobuf format, aren't available through the proxy.

---

📝 Examples & Use Cases

📖 Docs site: Example Configs guide describes each ready-to-run config under examples/.

Common Scenarios

• 🏢 Enterprise environments - Scrape metrics across firewall boundaries
• ☁️ Multi-cloud deployments - Bridge different network segments
• 🔒 Secure environments - Monitor internal services without opening inbound ports
• 🌐 Federation - Scrape existing Prometheus instances via /federate endpoint
• 🚀 Kubernetes - Monitor services across clusters or namespaces

Example Configurations

Use Case	Configuration
Basic setup	examples/simple.conf
Multiple apps	examples/myapps.conf
Metric filtering	examples/agent-filters.conf
Dynamic discovery	examples/discovery-targets.conf
TLS (no mutual auth)	examples/tls-no-mutual-auth.conf
TLS (with mutual auth)	examples/tls-with-mutual-auth.conf
Prometheus federation	examples/federate.conf
Nginx reverse proxy	nginx/nginx-proxy.conf

Advanced Use Cases

📖 Docs site: Advanced Topics guide covers the Nginx reverse-proxy setup, federation, and other advanced configurations.

Prometheus Federation

Scrape an existing Prometheus instance via the /federate endpoint:

agent.pathConfigs: [{
  name: "Federated Prometheus"
  path: "federated_metrics"
  url: "http://prometheus-server:9090/federate?match[]={__name__=~\"job:.*\"}"
}]

This leverages the existing service discovery features already built into Prometheus.

Another service discovery example config can be found in federate.conf.

Nginx Reverse Proxy

To use with Nginx as a reverse proxy, disable the transport filter on both Proxy and Agent:

java -jar prometheus-proxy.jar --tf_disabled
java -jar prometheus-agent.jar --tf_disabled --config myconfig.conf

An example nginx conf file is here, and an example agent/proxy conf file is here

⚠️ Note: With transportFilterDisabled, agent disconnections aren't immediately detected. Agent contexts on the proxy are removed after inactivity timeout (default: 1 minute, controlled by proxy.internal.maxAgentInactivitySecs).

gRPC Reflection

gRPC Reflection is disabled by default (see the security note below). Enable it for debugging and tooling with proxy.reflectionDisabled=false in the config file, -Dproxy.reflectionDisabled=false on the command line, or REFLECTION_DISABLED=false.

Test with grpcurl:

# List available services
grpcurl -plaintext localhost:50051 list

# Describe a service
grpcurl -plaintext localhost:50051 describe ProxyService

# When the proxy requires agent tokens, present one
grpcurl -plaintext -H 'agent-token: <token>' localhost:50051 list

If you use grpcurl -plaintext option, make sure that you run the proxy in plaintext mode, i.e., do not define any TLS properties.

⚠️ Security Note: Reflection exposes the full gRPC API surface: every service, method, and message type. When agent authentication is configured (the agent token or per-agent identities), reflection calls need a valid token like every other call on the agent port, and fail with UNAUTHENTICATED without one. With no agent authentication, anyone who can reach the agent port (default 50051) can use it, so enable it only on a trusted network. Hiding the API shape is not a substitute for authentication: still pair the agent port with agent tokens, mutual TLS, and/or network segmentation.

🔐 Security & TLS

📖 Docs site: Security overview and the TLS Setup guide cover agent authentication, mutual TLS, and certificate management in depth.

TLS Configuration Details

Agents connect to a proxy using gRPC. gRPC supports TLS with or without mutual authentication. The necessary certificate and key file paths can be specified via CLI args, environment variables, and configuration file settings.

The gRPC docs describe how to set up TLS. The repo includes the certificates and keys necessary to test TLS support.

Running TLS without mutual authentication requires these settings:

• certChainFilePath and privateKeyFilePath on the proxy
• trustCertCollectionFilePath on the agent

Running TLS with mutual authentication requires these settings:

• certChainFilePath, privateKeyFilePath and trustCertCollectionFilePath on the proxy
• certChainFilePath, privateKeyFilePath and trustCertCollectionFilePath on the agent

Run a proxy and an agent with TLS (with mutual authentication) with:

# Proxy with TLS
java -jar prometheus-proxy.jar \
  --cert /path/to/server.crt \
  --key /path/to/server.key \
  --trust /path/to/ca.crt

# Agent with TLS
java -jar prometheus-agent.jar \
  --config myconfig.conf \
  --trust /path/to/ca.crt \
  --cert /path/to/client.crt \
  --key /path/to/client.key

Run a proxy and an agent with TLS (no mutual authentication) using the included testing certs and keys with:

java -jar prometheus-proxy.jar --config examples/tls-no-mutual-auth.conf

java -jar prometheus-agent.jar --config examples/tls-no-mutual-auth.conf

Run a proxy and an agent docker container with TLS (no mutual authentication) using the included testing certs and keys with:

docker run --rm -p 8082:8082 -p 8092:8092 -p 50440:50440 -p 8080:8080 \
    --mount type=bind,source="$(pwd)"/testing/certs,target=/app/testing/certs \
    --mount type=bind,source="$(pwd)"/examples/tls-no-mutual-auth.conf,target=/app/tls-no-mutual-auth.conf \
    --env PROXY_CONFIG=tls-no-mutual-auth.conf \
    --env ADMIN_ENABLED=true \
    --env METRICS_ENABLED=true \
    pambrose/prometheus-proxy:4.4.0

docker run --rm -p 8083:8083 -p 8093:8093 \
    --mount type=bind,source="$(pwd)"/testing/certs,target=/app/testing/certs \
    --mount type=bind,source="$(pwd)"/examples/tls-no-mutual-auth.conf,target=/app/tls-no-mutual-auth.conf \
    --env AGENT_CONFIG=tls-no-mutual-auth.conf \
    --env PROXY_HOSTNAME=mymachine.lan:50440 \
    --name docker-agent \
    pambrose/prometheus-agent:4.4.0

Note: The WORKDIR of the proxy and agent images is /app, so make sure to use /app as the base directory in the target for --mount options.

Agent Token Authentication

The proxy accepts agent gRPC connections with no application-level authentication by default — any process that can reach the agent port (default 50051) can register as an agent. Set a shared pre-shared token so the proxy rejects agents that do not present it:

• Proxy: --agent_token, AGENT_TOKEN, or proxy.agentToken
• Agent: --agent_token, AGENT_TOKEN, or agent.agentToken

Both sides must use the same value. When set, the agent attaches the token as a gRPC metadata header on every call and the proxy rejects any call with a missing or mismatched token (UNAUTHENTICATED). When the token is empty (the default), the open behavior is preserved and the proxy logs a startup warning — unless mutual TLS is configured, which already authenticates agents.

# Proxy requiring a token
java -jar prometheus-proxy.jar --agent_token "$AGENT_TOKEN"

# Agent presenting the token
java -jar prometheus-agent.jar --config myconfig.conf --agent_token "$AGENT_TOKEN"

The token is a lightweight, app-level control. For production, prefer mutual TLS (and/or restrict the agent port to trusted networks); the token complements, but does not replace, those controls. The value is never written to logs.

Per-Agent Identities and Path Authorization

A single shared agentToken authenticates agents but does not distinguish between them: any agent holding the token can register any path, including one already served by another agent. To scope agents, define named identities under proxy.auth, each with its own token and a list of allowed path glob patterns:

proxy {
  auth = [
    { name = team-a, token = "team-a-token", paths = ["team_a_*"] }
    { name = team-b, token = "team-b-token", paths = ["team_b_*"] }
    { name = infra,  token = "infra-token",  paths = [] }          # empty paths = may register any path
  ]
}

Each agent presents its identity's token the usual way (--agent_token, AGENT_TOKEN, or agent.agentToken); no agent-side change is needed. The proxy resolves the token to an identity and enforces, on every registerPath, that the requested path matches one of the identity's paths patterns. Patterns are single-segment globs where * matches any run of characters and ? matches one (e.g. team_a_*); an empty paths list authorizes every path. An unknown token is rejected with UNAUTHENTICATED; a path outside the identity's patterns fails registration with a "not authorized" reason. Because authorization is per-identity-per-path, consolidated mode (multiple agents serving one path) still works as long as each agent's identity permits the shared path.

proxy.auth is a list of objects, so it is set in the config file only — there is no equivalent CLI flag or environment variable. Identity names must be unique and tokens must be non-empty; the proxy fails fast at startup otherwise.

Migrating from a shared token

Setting proxy.auth does not disable a legacy proxy.agentToken. When both are present, the shared token is honored as an additional allow-all identity (and the proxy logs a warning that it is active), so you can adopt per-agent identities incrementally:

1. Add a proxy.auth entry per agent while leaving proxy.agentToken in place — existing agents keep connecting with the shared token.
2. Move each agent onto its own identity token one at a time.
3. Once every agent presents an identity token, remove proxy.agentToken to close the shared allow-all path.

Auth Header Forwarding

When Prometheus scrape configurations include basic_auth or bearer_token, the proxy forwards the Authorization header to the agent over the gRPC channel. If TLS is not configured, these credentials are transmitted in plaintext and could be intercepted on the network between the proxy and agent.

Enable TLS when forwarding auth headers:

# Proxy with TLS to protect forwarded credentials
java -jar prometheus-proxy.jar \
  --cert /path/to/server.crt \
  --key /path/to/server.key

# Agent with TLS
java -jar prometheus-agent.jar \
  --config myconfig.conf \
  --trust /path/to/ca.crt

The proxy logs a warning on the first request that includes an Authorization header when TLS is not enabled.

Scrape Port Access

The scrape port (default 8080, listening on 0.0.0.0) has no authentication. Anyone who can reach it can have the agents scrape their targets, with the request's own query string and Authorization header passed through, and, with service discovery on, list every path with its agents' names and hosts. With an exporter that takes its target as a query parameter, such as blackbox_exporter's /probe?target=…, that reaches any host the exporter can. Bind proxy.http.host to a private address and firewall the port so only Prometheus reaches it; see Scrape Port Access.

Scraping HTTPS Endpoints

For HTTPS scrape targets signed by a custom or private CA (e.g. an internal corporate CA), point the agent at a trust store that contains that CA so certificates are still validated:

java -jar prometheus-agent.jar \
  --https_truststore /etc/agent/truststore.jks \
  --https_truststore_password changeit \
  --config myconfig.conf

This is also configurable via the HTTPS_TRUST_STORE_PATH / HTTPS_TRUST_STORE_PASSWORD environment vars or the agent.http.trustStorePath / agent.http.trustStorePassword properties. An empty path uses the JDK default trust store. The trust store is process-wide — it applies to every HTTPS target the agent scrapes.

As a last resort you can disable certificate verification entirely with the TRUST_ALL_X509_CERTIFICATES environment var, the --trust_all_x509 CLI option, or the agent.http.enableTrustAllX509Certificates property:

java -jar prometheus-agent.jar --trust_all_x509 --config myconfig.conf

⚠️ Security Note: Only use --trust_all_x509 in development/testing environments. Its scope is process-global and all-or-nothing: it disables certificate validation for every HTTPS target the agent scrapes, and it takes precedence over --https_truststore. Prefer a custom trust store whenever possible.

Reporting a Vulnerability

Report a security problem privately, not in a public issue, pull request, or discussion: use GitHub's private vulnerability reporting. SECURITY.md lists what a useful report includes and which documented behavior is by design.

🔧 Troubleshooting

📖 Docs site: Troubleshooting guide is a symptom-driven reference covering more failure modes and fixes.

Common Issues

Agent can't connect to proxy:

• Verify proxy hostname and port
• Check firewall rules (agent needs outbound access to proxy port)
• Ensure proxy is running and listening

TLS connection failures:

• Verify certificate paths and file permissions
• Check certificate validity and chain
• Ensure clock synchronization between proxy and agent

Metrics not appearing:

• Verify agent path configuration matches Prometheus scrape paths
• Check agent logs for scraping errors
• Confirm target endpoints are accessible from agent

Performance issues:

• Increase max_concurrent_clients for high-throughput scenarios
• Tune HTTP client cache settings
• Consider splitting targets across multiple agents (a consolidated path is scraped through every agent that serves it, so extra agents on one path add redundancy, not throughput)

📖 Documentation

Full documentation is available at the Prometheus Proxy Documentation site.

KDoc API documentation is published at pambrose.github.io/prometheus-proxy/kdocs.

Note: Running on Kubernetes? Helm charts for the proxy and the agent are in charts/, and the Kubernetes deployment guide covers them along with ready-to-use proxy and agent manifests, standalone and sidecar agent patterns, gRPC exposure for remote agents, and Prometheus Operator (ServiceMonitor) integration. A config that holds secrets, such as proxy.auth tokens or credentials in an agent's target URLs, can come from a Secret (existingSecret) instead of the chart's ConfigMap.

Note: Deploying to production? The Running in Production guide pulls the security, reliability, and tuning settings into a single operational checklist.

Maven Central

The library is published to Maven Central under com.pambrose:prometheus-proxy:

// build.gradle.kts
repositories {
  mavenCentral()
}

dependencies {
  implementation("com.pambrose:prometheus-proxy:4.4.0")
}

📄 License

This project is licensed under the Apache License 2.0 - see License.txt for details.

Related Links

• Prometheus.io
• gRPC
• Typesafe Config