otp_debug_mcp

Status: 0.13.0. Releases. Tested from Claude Code against live nodes on Erlang/OTP 26 to 29, on the same machine, over TLS distribution and across machines, with a test suite and CI on every push. Still young: the TODO list below has the known gaps.

An MCP server that lets AI agents (Claude Code, Claude Desktop, Cursor, GitHub Copilot, …; see docs/clients.md) connect to a running Erlang/Elixir/OTP node and debug it: not just raw introspection primitives, but an encoded diagnostic playbook in the spirit of recon and Erlang in Anger.

Design

• Hidden-node attach. The server joins the target over Erlang distribution as a hidden node that accepts no connections itself (no listening port, nothing registered with epmd, and a random cookie of its own), so it requires zero changes to the target application: no dependency, no plug, no redeploy. It works on anything running on the BEAM (Erlang, Elixir, Gleam, LFE), unlike in-app servers such as Tidewave or extools.
• Diagnosis-first tool design. The agent's first call is diagnose, which runs a whole battery of checks and returns ranked findings with interpretations — instead of making the AI reconstruct BEAM operational knowledge from twenty process_info round-trips.
• recon injected at connect time, only where missing. recon's object code is loaded into the target over rpc, so recon-quality introspection works even on nodes that never shipped it. A target on an older OTP release than the tool can't load its compiled code, so there recon is compiled on the target from its abstract code (this needs the target's compiler application, which Elixir releases always include). If the target has its own recon, that copy is used as-is and never loaded over: replacing a loaded module and later purging it would kill processes running it, such as an operator's recon_trace session. connect reports which recon is in use.
• Safe by default. Every trace has hard caps on event count and duration and stops early if calls arrive too fast; every profile has a capped duration and a scope you choose; every rpc has a timeout; arbitrary remote_eval is disabled unless OTP_DEBUG_MCP_ALLOW_EVAL=true.
• Nothing changes the node's code without you. load_module asks you to approve every load in a dialog from your MCP client, even when you've allowed the tool in the client's permissions. A client that can't show the dialog gets a refusal.
• Tracing never disturbs someone else's. On OTP 27+ targets, trace_calls runs in its own isolated trace session (trace:session_create/3), so other tracing on the node (an operator's recon_trace, dbg, anything using erlang:trace/3) keeps running, even on the same function. On older targets it falls back to recon_trace, which turns off all tracing on the node when it starts and stops. There it first checks for other tracing and refuses, describing what it found, unless called with force: true. The bundled skill tells the agent to ask the user before forcing. profile and trace_port work the same way: their own sessions on OTP 27+, and a refusal on older nodes while anyone else traces.

Tools

Tool	Purpose
connect	Attach to a node (name + optional cookie) as a hidden node
diagnose	Full health battery, findings ranked worst-first
run_check	One check: vm_config (the VM's setup against what it runs on: schedulers against the container's CPU quota, and how often it's throttled; memory against the container's limit; limits in use; busy-waiting, debug logging), message_queues, call_chains, supervisor_restarts, binary_leak, memory_breakdown, allocator_fragmentation, process_count, atom_table, ets_tables, ports, native_memory, native_code, stale_code (old code still running after a hot load, modules unlike their .beam on the code path), logs (where the node's logs go, and whether a file keeps them), distribution (connections to other nodes: traffic, and bytes waiting against the busy limit), schedulers, scheduler_time (where scheduler time goes: Erlang code, garbage collection, ports, I/O polling, busy-waiting), long_schedules, socket_traffic, top_reductions
recent_errors	The errors the node logs, grouped by who failed and why: crashes with their stack and the reports they caused, error and warning messages with where they were logged, and the errors in the node's log files from before, and processes that crashed calling each other (a deadlock a call timeout ended); listens for a window, or in the background
crash_dump	Why a node died, from its erl_crash.dump: what the slogan means, the process that was running, the biggest processes and mailboxes with their stacks, the first messages of the longest mailbox, memory, ETS and ports; no connection needed
process_info	Deep-inspect one process: recon:info, an age estimate, its supervisor's restart history, optional sys:get_state
supervision_tree	Render an application's supervision tree, with each supervisor's strategy and restart history
ets_table	What's in one ETS table: its type, protection, owner and heir, its first rows, and a bounded scan for the average and largest rows and, for a bag, the most repeated keys

The diagnostic checks live in lib/otp_debug_mcp/checks.ex; each returns structured findings plus an interpretation telling the agent what the numbers mean and what to look at next. supervisor_restarts finds crash loops: it reads every supervisor's own restart history (including DynamicSupervisor), flags supervisors one crash away from giving up, and names the children restarting right now, with the exit reason and stack trace of one seen to die. call_chains finds who waits on whom: deadlocks (processes calling each other), chains of calls, and the processes most waited on.

Trying a fix on the live node

Compile the fix as usual (mix compile, rebar3 compile, erlc) and ask the agent to load it:

I've fixed payments_gateway.erl and compiled it to _build/default/lib/payments/ebin. Load it onto payments@staging, then check that the crashes stop.

load_module reads the .beam file and asks you, in your client, to approve that exact code on that node. The dialog shows the md5 and path of the new code and of the version running now. In Claude Code it looks like this:

MCP server "otp-debug" requests your input
Load new code for payments_gateway on payments@staging?
New code: md5 2097d2e9edaf7d52e3bf00103def52e3, /home/you/payments/_build/...
Running now: md5 247da95dfc69d91ae70a39b0ee346b88, /opt/payments/lib/payments-1.4.0/ebin/...
❯ * Load it: ☐
      Every process at once, until the node restarts
  Accept    Decline

Tick the box and accept to load it. Declining, or pressing Esc, loads nothing, and the agent is told not to ask again unless you say so. If the file changes after the dialog appears, you're asked again. The approval can't come from the agent: it's your answer to a form your client shows you, and the tool can't be called around it.

The load works like nl/1 in a shell. Every process switches to the new code on its next call into the module, and the node's own files aren't changed, so a restart brings back the old version. load_module(revert: "payments_gateway") goes back to the version on the node's code path, also only once you approve. The tool doesn't purge code that's still running, so it never kills a process. A process still in a function of an older version blocks the next load of that module, and the tool says which processes those are. It doesn't replace OTP's own modules, like lists or gen_server.

The dialog needs a client that can ask you mid-call: one using MCP 2026-07-28 with form elicitation, as Claude Code does. Other clients get a refusal and nothing is loaded. In claude -p there's nobody to answer, so loads are always refused. Walkthrough 4 is a whole recorded session of it.

Asking before every step

To approve the investigation step by step, start the server with OTP_DEBUG_MCP_CONFIRM:

Value	The server asks before
off (the default)	nothing but load_module's loads
changes	each step that does something on the node beyond reading it: connecting (which may load recon there), diagnose (it forces garbage collections and briefly turns on microstate accounting and a trace session), checks that do the same, tracing, profiling, recent_errors, telemetry, telemetry_handlers, bandit, cowboy and shackle (which attach a handler while they listen), mnesia (which subscribes to its events), benchmark (which runs the functions it measures), postgrex with server (which opens a connection to the database), process_info with the process's state, riak_core with crdts (it asks the vnodes for their state), cluster, and remote_eval
all	every step, reads included

In Claude Code:

claude mcp add otp-debug --scope user -e OTP_DEBUG_MCP_CONFIRM=all -- "$server" start

or "env": { "OTP_DEBUG_MCP_CONFIRM": "all" } in a client's JSON config. Each step then shows a dialog first: the tool, the node, what it does there, and its arguments.

MCP server "otp-debug" requests your input
Run diagnose on payments@prod-2?
Runs every check: reads the node's processes, memory, ports, ETS, logs and
code; forces a garbage collection in every process (binary_leak); ...
  Accept    Decline

Accept runs it. Decline, or Esc, runs nothing, and the agent is told not to run it again but to ask you how to go on. The answer counts only for that call, with those arguments. The setting is the server's: unlike a client's own permission prompt, an "always allow" in the client doesn't skip it, and the dialog says what the step does to the node. It needs a client that can ask you mid-call (MCP 2026-07-28 form elicitation, as Claude Code does): other clients, and claude -p, where nobody can answer, get a refusal and nothing runs.

Clients also have their own per-call prompts, which work alongside it. Claude Code asks before each MCP tool call unless you allowed the tool: answer "Yes" rather than "don't ask again", or put "ask": ["mcp__otp-debug"] under permissions in its settings to keep asking. For other clients, see docs/clients.md.

To change the setting, re-register the server with the new value and reconnect it. It can't be changed while the server runs: it's read from the environment the client starts the server with, so the agent can't turn off the approvals that check it.

claude mcp remove otp-debug -s user
claude mcp add otp-debug --scope user -e OTP_DEBUG_MCP_CONFIRM=changes -- "$server" start

Then in Claude Code, /mcp, choose otp-debug, and Reconnect (or restart Claude Code). The new server process doesn't keep the old one's connection, so the agent connects to the node again. In other clients, edit the env block of the server's config and restart or reconnect it.

talks/step-by-step.pptx is a talk, with speaker notes, that goes through one investigation (the crash loop of scenario 6) dialog by dialog with OTP_DEBUG_MCP_CONFIRM=all, declining one step.

Setup

Install with Claude Code

Claude Code can do the whole installation: download the release for your machine, verify it, register it and install the skill. It asks before each command, so you see everything it does. In a terminal:

claude "Install otp_debug_mcp from https://github.com/ayarodionov/otp_debug_mcp \
by following its README: download the latest release for this machine, \
verify its checksum, register it with Claude Code for all projects \
(replacing any existing otp-debug registration), and install the skill. \
Check that each step worked."

To build from source instead, ask for that, and say where to clone it. Then quit Claude Code and start it again: it loads MCP servers when it starts, and /mcp should list otp-debug as connected. Or install by hand, as below.

Download a release

Releases bundle their own Erlang runtime, so nothing else needs installing. They're built for:

• macOS on Apple silicon (macos-arm64)
• Linux on x86_64 and arm64 (linux-x86_64, linux-arm64), with glibc from Ubuntu 22.04 or later and OpenSSL 3 (libcrypto.so.3)

Set target to macos-arm64, linux-x86_64 or linux-arm64:

vsn=0.13.0
target=macos-arm64
base=https://github.com/ayarodionov/otp_debug_mcp/releases/download/v$vsn
curl -fLO "$base/otp_debug_mcp-$vsn-$target.tar.gz"
curl -fLO "$base/otp_debug_mcp-$vsn-$target.tar.gz.sha256"
shasum -a 256 -c "otp_debug_mcp-$vsn-$target.tar.gz.sha256"
mkdir -p ~/.local && tar -xzf "otp_debug_mcp-$vsn-$target.tar.gz" -C ~/.local
ls ~/.local/otp_debug_mcp/bin

shasum should print OK, and ls should list otp_debug_mcp.

The server is then ~/.local/otp_debug_mcp/bin/otp_debug_mcp. The binaries aren't signed: on macOS, an archive downloaded with a browser rather than curl is quarantined, and the server won't start until you run xattr -dr com.apple.quarantine ~/.local/otp_debug_mcp.

Build from source

Needs Elixir 1.19 and Erlang/OTP 27 or later. Clone the repository anywhere you like, then build in it:

git clone https://github.com/ayarodionov/otp_debug_mcp.git
cd otp_debug_mcp
mix deps.get
MIX_ENV=prod mix release

The server is then _build/prod/rel/otp_debug_mcp/bin/otp_debug_mcp in that directory.

Or use the Docker image

For nodes in Docker containers, the server can run in a container beside them, from the image published with each release:

docker pull ghcr.io/ayarodionov/otp_debug_mcp:0.13.0

Targets in Docker containers says how to register it, so that Claude Code starts it with docker run on the app container's network. For nodes on your machine, use a downloaded release or a build from source.

Register it with Claude Code

First say where the server and the skill are. For a downloaded release:

server=~/.local/otp_debug_mcp/bin/otp_debug_mcp
skill=~/.local/otp_debug_mcp/skills/otp-debug

For a build from source, set checkout to the directory you cloned and built in, replacing /path/to/otp_debug_mcp:

checkout=/path/to/otp_debug_mcp
server=$checkout/_build/prod/rel/otp_debug_mcp/bin/otp_debug_mcp
skill=$checkout/skills/otp-debug

Then register the server (--scope user makes it available in every project) and install the skill, so Claude follows the debugging playbook:

claude mcp add otp-debug --scope user -- "$server" start
mkdir -p ~/.claude/skills && cp -R "$skill" ~/.claude/skills/

claude mcp get otp-debug should show the server's path, and /mcp in Claude Code should list otp-debug as connected. If claude mcp add says otp-debug already exists, an earlier registration is in the way: remove it with claude mcp remove otp-debug --scope user and add it again. A build from source runs whatever was built last, so run mix release again after updating the checkout.

Or as a .mcp.json entry. JSON expands neither ~ nor variables, so command must be the server's full path, the one server holds above (echo "$server" prints it): /Users/you/.local/otp_debug_mcp/bin/otp_debug_mcp for a downloaded release, or /path/to/otp_debug_mcp/_build/prod/rel/otp_debug_mcp/bin/otp_debug_mcp for a build from source. For a downloaded release:

{
  "mcpServers": {
    "otp-debug": {
      "command": "/Users/you/.local/otp_debug_mcp/bin/otp_debug_mcp",
      "args": ["start"],
      "env": { "OTP_DEBUG_MCP_ALLOW_EVAL": "false" }
    }
  }
}

For Cursor, GitHub Copilot or Claude Desktop, see docs/clients.md.

Notes:

• Name mode is automatic. connect starts distribution on demand in the mode the target uses: shortnames for app@host (the Mix release default), longnames for [email protected] or [email protected]. Connecting to a target in the other mode switches modes, dropping the previous connection.
• Reconnects. If the target restarts, the next tool call reconnects and re-injects recon. If it's down, tools report connection_lost.
• Cookie. Without a cookie argument, the server uses the OTP_DEBUG_MCP_COOKIE environment variable if it's set, which keeps a production cookie out of the conversation, and otherwise ~/.erlang.cookie, which matches iex --sname/erl -sname. Mix releases generate their own cookie (RELEASE_COOKIE, or releases/COOKIE).
• Production nodes. docs/production.md covers firewalls and fixed distribution ports, targets that listen only on loopback (with an SSH tunnel recipe), and TLS distribution.
• Docker. For a node in a container, run the server in a container beside it: no changes to the app, no ports published. Or publish the node's port. Targets in Docker containers has both, and a prompt that has Claude Code set it up.
• Kubernetes. For nodes in pods, bin/otp-debug-k8s in the release runs the server as a pod in the app's namespace for each session, with the cookie from the app's Secret. Targets in Kubernetes has the set-up, the permissions it needs, and port forwarding.
• Pids look the same everywhere. Every tool prints the target's pids and ports the way the target itself does (<0.107.0>), matching trace output and remote_eval, and process_info accepts them as given.
• Erlang or Elixir syntax. Terms from a plain Erlang node are shown in Erlang syntax, from an Elixir node in Elixir syntax.
• stdout is the protocol stream. All logging goes to stderr, and IO from the target during rpc calls and traces is redirected away from stdout.

Then, in Claude Code:

Connect to [email protected] and find out why memory keeps growing.

Stop, restart or remove it

Claude Code starts the server when a session starts and stops it when the session ends, so there's no server to stop by hand. In a session, /mcp → otp-debug has:

• Reconnect: stops the server and starts a fresh one, which forgets its connections and snapshots. Use it after installing a new release or running mix release again, so the session runs the new build, or if the server stops responding.
• Disable: stops the server and keeps it off in this project. Enable, in the same place, brings it back.

Stopping the server closes its connections to the target nodes. They carry on as before, except that recon modules the tool loaded stay loaded until the node restarts. Claude Code can't run /mcp for you, so use the menu for these two.

To remove it from your machine, take out what the install put in:

claude mcp remove otp-debug --scope user
rm -rf ~/.claude/skills/otp-debug

claude mcp list should no longer show otp-debug. If you also registered it for one project (--scope project or local, or an entry in a .mcp.json), run claude mcp remove otp-debug in that project too. Then delete the server: for a downloaded release, that's

rm -rf ~/.local/otp_debug_mcp

and for a build from source, the directory you cloned and built in.

Like any Erlang node, the server starts epmd, the port-mapper daemon on port 4369, if it isn't running yet, and epmd keeps running after the server stops. It's harmless and other Erlang nodes use it too. If pgrep -fl epmd shows it running from ~/.local/otp_debug_mcp, stop it before deleting that directory:

~/.local/otp_debug_mcp/erts-*/bin/epmd -kill

It refuses with "Killing not allowed - living nodes in database" while any node is registered (epmd -names lists them); stop those first, or leave epmd running.

For Cursor, GitHub Copilot or Claude Desktop, also take otp-debug out of their MCP config, and delete the playbook files docs/clients.md had you create (.cursor/rules/otp-debug.mdc, .github/instructions/otp-debug.instructions.md).

Claude Code can do the removal too, asking before each command:

claude "Remove otp_debug_mcp from this machine, following the README at \
https://github.com/ayarodionov/otp_debug_mcp: remove the otp-debug MCP \
server from every Claude Code scope it's registered in, remove the \
otp-debug skill, stop epmd if it runs from the release and no nodes are \
registered, and delete the installed release. If it was built from \
source, show me the checkout and ask before deleting it. Check that \
each step worked."

That session starts its own otp-debug server from the release it's deleting, which keeps running until you quit the session; that's harmless, as nothing else uses it.

Or to upgrade to the latest release:

claude "Upgrade otp_debug_mcp to the latest release from \
https://github.com/ayarodionov/otp_debug_mcp by following its README: \
download the release for this machine, verify its checksum, replace the \
installed one in ~/.local/otp_debug_mcp, and copy its skill to \
~/.claude/skills again. Check that each step worked."

Then use /mcp → otp-debug → Reconnect in any session that's already running, so it uses the new release.

Typical agent workflow

1. connect(node: "[email protected]"); if the node is in a cluster, it says so, and cluster() compares every node to find the one that differs
2. diagnose() → ranked findings
3. Drill in: process_info on offenders, supervision_tree for context, trace_calls to see live traffic
4. snapshot(take) → wait → snapshot(diff) to confirm leaks/trends

The Claude Code skill installed in Setup, skills/otp-debug/SKILL.md, encodes this workflow, so Claude follows it automatically.

Usage scenarios

docs/scenarios.md walks through each of these with an example prompt, the tools involved, and what to watch out for:

• Production problems: memory growth, slow requests, high CPU, stuck processes and call timeouts, approaching system limits, crash loops and intermittent failures, errors only the logs show, a node that died, one bad node in a cluster, incident triage, NIFs that block a scheduler or leak native memory, leaking sockets and external programs, slow peers, garbage collection eating the CPU, traffic between nodes, a web server refusing or stalling connections, a riak_core cluster in trouble
• Understanding and verifying: tracing a runtime problem to the code in your repo, seeing what a function really receives, what's in an ETS table, verifying a fix or a deploy, a deploy or hot load that didn't take, trying a fix on the live node, code that macros generated, watching a load test, learning an unfamiliar system
• Other targets: Erlang products you didn't write (message brokers, databases), and your own node during development

It ends with a checklist for production use and the scenarios the tool doesn't support yet. talks/three-scenarios.pptx is a 15-minute talk, with speaker notes, on three of them: a crash loop and its fix tried on the live node, errors only the logs show, and a node that died.

docs/walkthroughs.md has whole sessions to follow along with on the demo nodes below: the prompt, every tool call with its real result, and the conclusion, for a mailbox that never drains, a crash loop, one bad node in a cluster, trying the crash loop's fix on the node with load_module, a bug that exists only in code a macro generated, a Phoenix route shadowed by one resources generates, NIFs that block a scheduler and leak native memory, leaking sockets and external programs with a slow peer, errors only the logs show, a node that died, from its crash dump, two gen_servers that deadlock, a Phoenix API seen through its telemetry, failures that nothing logs, what's in an ETS table, CPU spent collecting garbage, a fix deployed but never loaded, a link between nodes saturated, a Bandit server out of connections, a Cowboy server whose acceptors wait for room, a Yaws server starting a session on every request, a shackle pool with one connection held up, a Ranch server whose dead connections keep new clients out, a LiveView dashboard whose assigns keep every order, telemetry handlers piling up, Mnesia transactions waiting on one record, a Postgrex pool held up by a payment call, profiling and tuning a slow function, an EMQX broker with a slow consumer and a client-id clash, a RabbitMQ broker whose memory alarm blocks its publishers, a service throttled by its container's CPU quota, a gen_server whose callers wait behind one slow request, quotes slowed by Nagle's algorithm, an ejabberd server with a bot that reads slowly, its archive on Mnesia and an offline queue full, a riak_core cluster with a member down and a vnode swamped, and riak_dt sets whose actors pile up.

docs/plausible-but-wrong.md runs six "Plausible but wrong" examples from the books Learning Erlang from Claude and Learning Elixir from Claude, each on a live node, and shows an agent given only the symptom finding each one with these tools.

Demo targets

examples/ has demo nodes in each BEAM language, each with a bug for the tool to find. Each directory's run.sh builds and starts them in the background, on this host with -sname, and run.sh stop stops them:

Language	Directory	Demos
Elixir	examples/elixir	a mailbox that never drains, a crash loop, a cluster with one bad node, a billing service whose macro DSL records every charge twice, a Phoenix API whose search route is shadowed by one resources generates, a LiveView dashboard whose assigns keep every order and whose Refresh blocks it, a Bandit API out of connections, a Cowboy API whose slow route leaves no room for the fast ones, a Yaws API whose cookie sessions pile up, a shackle pool where one slow request holds up a connection, a point-of-sale server on Ranch whose dead connections leave no room for new tills, a checkout slowed by telemetry handlers it never detaches, product pages waiting for Postgrex connections held through a payment call, a riak_core cluster with a member down and a hot partition, riak_dt wishlists whose actors pile up, and a Mix release with its own cookie
Erlang	examples/erlang	a mailbox that never drains, an OTP application in a crash loop, a session store leaking atoms and ETS rows, a cluster with one bad node, a C NIF library that blocks a scheduler and leaks native memory, a shop that leaks sockets and external programs and fills a slow peer's socket, an orders service whose errors only its logs show, a node that runs out of memory and leaves a crash dump, two gen_servers that deadlock, report workers whose CPU goes to garbage collection, a fix deployed but never loaded, and a node flooding the link to another
Gleam	examples/gleam	a mailbox that never drains, a supervised actor whose let assert panics in a loop, and a job runner leaking processes
LFE	examples/lfe	a mailbox that never drains, an OTP application in a crash loop, and an index whose slices pin whole binaries
Sockets	examples/sockets	An Erlang quotes service in Docker, with a request written in two pieces with Nagle on (41 ms a quote), an collector flooded by its agent, and an export blocked by a slow reader with no send timeout

examples/books has six "Plausible but wrong" examples from the books Learning Erlang from Claude and Learning Elixir from Claude, in Erlang and Elixir; docs/plausible-but-wrong.md shows what an agent found on each, and talks/plausible-but-wrong.pptx is a 7-minute talk about it, with speaker notes.

Point Claude Code at one without saying what's wrong:

shop_ex@yourhost is getting slower and using more memory; find out why.

Headless Claude Code sessions, given only a prompt like that and no source, found each kind of bug: the crash loop's cause, file and line from diagnose alone; the cluster's bad node through cluster, pointed at a healthy one; the atom leak, the process leak and the pinned binaries through diagnose. CI smoke-tests the release against the shop demo in each language.

Testing

mix test

The tests start real target nodes with OTP's peer module and connect to them over distribution as hidden nodes, exactly as against a live system: connection and name modes, reconnects, recon injection, every check, snapshots, both tracing backends, and the tools' JSON output. Fixture code for the targets is in test/support/fixtures.ex. Tests tagged old_otp need a node running an older OTP release, which the Docker recipe below provides; see test/otp_debug_mcp/old_otp_test.exs. Tests tagged new_otp need one running a newer release than the tool's, in the same way; see test/otp_debug_mcp/new_otp_test.exs. The tests run a Yaws server too, fetched from GitHub, whose build compiles a PAM driver: on Debian and Ubuntu, install libpam0g-dev first. The Postgrex tests start a throwaway PostgreSQL, with initdb and pg_ctl from the PATH or /usr/lib/postgresql/*/bin; without them they're skipped. The EMQX tests need an EMQX node, given by EMQX_NODE and EMQX_COOKIE (see test/otp_debug_mcp/emqx_test.exs; CI runs one in Docker), and the ejabberd tests an ejabberd node, given by EJABBERD_NODE and EJABBERD_COOKIE (see test/otp_debug_mcp/ejabberd_test.exs; CI runs one in Docker too), and the RabbitMQ tests a RabbitMQ node, given by RABBITMQ_NODE and RABBITMQ_COOKIE (see test/otp_debug_mcp/rabbitmq_test.exs; CI runs one in Docker).

The whole suite also runs over TLS distribution, peers included, which adds a test that the target sees the connection as TLS:

CONF=$(scripts/tls_dist_certs.sh /tmp/tls)
ERL_FLAGS="-proto_dist inet_tls -ssl_dist_optfile $CONF" mix test

CI (.github/workflows/ci.yml) runs the tests on OTP 27 and 28, again over TLS distribution, the old_otp tests against an OTP 26 container, the new_otp tests against an OTP 29 one, and the smoke test below against the built release.

scripts/mcp_smoke.py is an end-to-end check that drives the release over MCP stdio the way a client does, against a demo node with a real bug:

examples/elixir/run.sh shop
python3 scripts/mcp_smoke.py \
  _build/prod/rel/otp_debug_mcp/bin/otp_debug_mcp "shop_ex@$(hostname -s)" Shop.PriceCache

It fails on any non-JSON line on stdout, a tool error, an empty trace, or an exit status other than 0 when stdin closes.

scripts/mcp_call.py runs an arbitrary sequence of tool calls, with shell steps in between, e.g. to restart the target. Against the Elixir release demo, which has its own cookie:

examples/elixir/run.sh release
REL=examples/elixir/demo_shop/_build/prod/rel/demo_shop
python3 scripts/mcp_call.py _build/prod/rel/otp_debug_mcp/bin/otp_debug_mcp "[
  [\"connect\", {\"node\": \"demo_shop@$(hostname -s)\", \"cookie\": \"$(cat $REL/releases/COOKIE)\"}],
  [\"supervision_tree\", {\"app\": \"demo_shop\"}],
  [\"shell\", \"$REL/bin/demo_shop stop; sleep 2; $REL/bin/demo_shop daemon; sleep 4\"],
  [\"run_check\", {\"check\": \"top_reductions\"}]
]"

Verified so far: every tool through MCP stdio; shortnames, longnames (IP and FQDN) and switching between them in one session; Mix release cookies; reconnect after a target restart; targets with their own recon (including an old one missing functions) keep their running recon sessions across repeat connects and checks; on OTP 28, trace_calls traces the same function as an operator's recon_trace session without disturbing it; on an OTP 26 target, recon is compiled on the target and every check works, and trace_calls refuses while dbg is tracing unless forced; the whole test suite over TLS distribution; a target on another Mac over the local network (OTP 29, newer than the tool's OTP 28), reached by IP with a fixed distribution port, where the smoke test and every tool passed and the target saw the tool only as a hidden node; and, between the same two Macs, both recipes in docs/production.md: an SSH tunnel to a target listening only on loopback, and TLS distribution, which refused plain-TCP connections and certificates from another CA.

scripts/recon_target.exs is a target with its own recon and an operator's recon_trace session open, logging once a second whether it's still alive, for checking the tool never disturbs it:

STATUS_FILE=/tmp/status.log elixir --sname rt -pa _build/prod/lib/recon/ebin \
  --no-halt scripts/recon_target.exs &
python3 scripts/mcp_call.py _build/prod/rel/otp_debug_mcp/bin/otp_debug_mcp "[
  [\"connect\", {\"node\": \"rt@$(hostname -s)\"}], [\"diagnose\", {}],
  [\"connect\", {\"node\": \"rt@$(hostname -s)\"}]
]"
tail -1 /tmp/status.log

The last line should read operator_trace=alive.

An older OTP release runs best in Docker. Distribution through a published port works without epmd: the target listens on a fixed port, and the tool connects to that port without listening itself:

docker run -d --rm --name otp26 -p 9100:9100 --cpus=1 erlang:26-slim \
  erl +S 4:4 -noinput -name [email protected] -setcookie testcookie -erl_epmd_port 9100 \
  -eval 'spawn(fun() -> timer:apply_interval(100, queue, new, []), receive after infinity -> ok end end).'
ERL_FLAGS="-erl_epmd_port 9100" python3 scripts/mcp_call.py \
  _build/prod/rel/otp_debug_mcp/bin/otp_debug_mcp '[
  ["connect", {"node": "[email protected]", "cookie": "testcookie"}],
  ["diagnose", {}],
  ["trace_calls", {"module": "queue", "function": "new", "max_events": 4}]
]'

Blind sessions

The walkthroughs are also a regression suite: scripts/blind_eval.py runs each one as it was recorded. It starts the demo, gives a headless claude -p session, in an empty directory, only the symptom and the otp-debug tools, stops the demo, and checks that the session's conclusion names the cause (the patterns in test/blind/cases.json). It needs the Claude Code CLI, logged in, and costs what the sessions do; it isn't part of CI.

python3 scripts/blind_eval.py --list                    # the cases
python3 scripts/blind_eval.py --build --skip docker,postgres
python3 scripts/blind_eval.py --cases inventory,ranch   # some of them

Transcripts, each session's answer and a report (report.md, report.json: pass or fail, time, tool calls, what was missing) go to blind-results/<time>/. A change to a tool that makes an agent's job harder shows there as a failure, or as more tool calls and time.

Releasing

1. Set the new version in mix.exs and in the Dockerfile's ARG VSN, and add a ## <version> section for it at the top of CHANGELOG.md.
2. Commit, then tag and push: git tag v<version> && git push origin v<version>.

The Release workflow builds and smoke-tests each platform's archive, checks the tag matches mix.exs, and publishes the GitHub release with the changelog section as its notes. Then the Image workflow builds the Docker image from the published Linux archives, for amd64 and arm64, smoke-tests it against a node in another container, and pushes it to ghcr.io/ayarodionov/otp_debug_mcp as <version> and latest. Run the Release workflow by hand (gh workflow run Release) to build and test without publishing, and the Image workflow (gh workflow run Image -f version=<version>) to build an earlier release's image.

Status / TODO

Open items:

• Targets on an older OTP release than the tool, without the compiler application (common in slim rebar3 releases), get no recon: ship recon precompiled for older releases
• trace_calls on OTP 26 and earlier still goes through recon_trace, which ends other tracing (with permission): a small tracer of our own