Works with what you already run
Whatever you built it with, PromptPotter meets it where it already runs:
| You built it with | How PromptPotter meets it |
|---|
| An agent framework — LangGraph, LlamaIndex, CrewAI, Microsoft Agent Framework / Semantic Kernel, Google ADK, AWS Strands Agents, the OpenAI Agents SDK, Haystack, Spring AI, or plain FastAPI | Put it behind three endpoints — POST /matches, GET /pipeline, GET /status — and the knobs your /pipeline declares become what is tuned (backend integration) |
| A managed agent service — AWS Bedrock Agents, Google Vertex AI Agent Engine, Azure AI Foundry | Your deployed agent already has an endpoint; a thin adapter in front of it answers the same three |
| A workflow builder — n8n, Dify, Langflow, Flowise | The same three endpoints in front of the workflow's API; the prompts and node settings it exposes become the knobs |
| DSPy | PromptPotterOpt, where you would reach for GEPA (dspy-optimizer) |
| An agent that works in a sandbox | Each candidate runs as a Harbor trial, graded by the task's own checker; the agent's trajectory is the evidence for the next mutation |
| Langfuse / MLflow | Every round fans out as traces; the winner lands in MLflow's Prompt Registry (observability) |
| OpenRouter, OpenAI, Anthropic, Groq | Any of them, for your pipeline and for the optimizer; OpenRouter is the default |
What you get back
export.json — the winning prompt and settings by field name, with the evidence behind them. We write that file and provide a reader; we never load, host, route or hot-swap anything in your production path.
What it does differently
Prompt optimization that stops paying for losers. Every measurement costs money, so the whole design is most fitness per dollar.
Four more claims follow, with the screens that back them.
It moves the pipeline, not just the words
Model, temperature, node thresholds, the output schema — whatever your pipeline.yaml declares tunable — mutate and are selected jointly with the prompt. In the round replayed above, two of the three mutations were to a schema field description, not to the prompt. Reading the run card →
The budget goes where candidates separate
Scoring goes to the samples that tell candidates apart; anything already measured is replayed, never re-bought.
It does not certify its own winner
Scores are a subset-invariant ability (Rasch θ), and each round says when its lift is noise. What a winner's number may claim →
A campaign is an object on disk
Not a script that has to finish. Each round is a JSON file the app renders as a scoreboard; pause, resume, rewind to any round, or fork a sibling and compare. Persistence and recovery →
The thread, the run card, the optimizer's own loop, the sample leaderboard, one measured cell opened. 📺 Longer: Live #2 — a Swiss-invoice campaign, run raw.
Quickstart
Python 3.13+ and an OpenRouter key — the optimizer's default provider.
pip install "promptpotter[all]"
python -m promptpotter new tickets.csv --set task_description="classify support tickets by urgency"
python -m promptpotter resume
Bring a labeled dataset — input/output pairs as any delimited text, JSON, JSONL or a spreadsheet (.xlsx, .xls, .ods) — and say what the job is. No key set yet? new offers to write one on first run. resume picks up wherever the last run stopped: Ctrl+C pauses a campaign rather than losing it.
The DSPy and Harbor connectors are opt-in: pip install "promptpotter[dspy]" or "promptpotter[harbor]". Only these bring LiteLLM in.
[!IMPORTANT]
new needs a pipeline backend to score against — the default is TermNorm on :8000, and a campaign refuses to start without one (setting it up by hand).
Shortest path: in Claude Code, type /potter-run. It downloads and starts that backend for you, audits your setup, picks a dataset, runs init, and reads the rounds back to you as they land.
To watch it in the app shown above:
python -X utf8 -m uvicorn promptpotter.main:app --port 8001 # then open http://localhost:8001/
from promptpotter.presentation.teleprompter import PromptPotterOpt
optimizer = PromptPotterOpt(metric=my_metric, dataset_name="my-task")
compiled = optimizer.compile(my_program, trainset=trainset)
What you keep and what you trade away that way: dspy-optimizer.md.
Six chapters from install to troubleshooting: the manual.
Five entry points
One campaign, driven from wherever the work already happens:
| |
|---|
| Webapp | The browser control plane on this page, at the domain root — app.promptpotter.com hosted, http://localhost:8001/ local. Chat-first: talk to the Potter and watch it work inline, with a button whenever a decision is yours. Ships as a reusable chat-app template (spec). |
/potter-run | Claude Code skill — an agent launches, supervises and diagnoses a full campaign from your editor. |
| CLI | python -m promptpotter new <name> / resume. |
| REST API | Every control verb the webapp issues, for when another system drives the campaign. |
| Embedded in Python | A host program drives one campaign inside its own event loop; the DSPy optimizer is the simple-case entry. |
A sixth — PromptPotter as a tool another agent calls (MCP) — is open for discussion in #27.
How the search works
Every round generates, scores and critiques (L1). When L1 stalls, L2 rewrites how it searches; when L2 stalls, L3 rewrites the plan. The loop exits on a stopping rule — goal, budget or plateau — not a fixed trial count. The loop →
It searches a tree, not a trail. When a branch is spent, a UCB rule picks the ancestor to re-expand, so the search climbs a different hill instead of stalling — AlphaZero-shaped MCTS over the lineage, with deterministic evaluation in place of rollouts (the mapping).
How it compares to GEPA and AlphaEvolve
PromptPotter belongs to the LLM-driven evolution family — an LLM proposes variants, a scorer ranks them, the winners breed. On the prompt side that family is GEPA, MIPROv2, PromptWizard and CAPO; on the code side AlphaEvolve and OpenEvolve. GEPA evolves the instruction text of a DSPy program, AlphaEvolve evolves source code, PromptPotter evolves a whole declared pipeline jointly with the prompt — a different target, not a missing feature.
The table is capability, not a benchmark result:
| Capability | GEPA | AlphaEvolve | PromptPotter |
|---|
| Tunes the whole pipeline — model, temperature and node thresholds evolve alongside the prompt | 🟡 | 🟡 | 🟢 |
| Stops losers early — scores share one scale, so nobody wins by drawing an easier set | 🟡 | 🟡 | 🟢 |
| Every measurement is priced — a cache-served result keeps its full cost on the ledger | 🔴 | — | 🟢 |
| Self-healing — an invalid proposal is caught and taught to a different layer, not just discarded | 🟡 | 🟡 | 🟢 |
| Carries knowledge across runs — parameter impact, sample difficulty and failure patterns survive the campaign | 🟡 | 🟢 | 🟢 |
| Open and inspectable — a browser control plane, and a campaign you pause, resume, rewind or fork | 🟡 | 🔴 | 🟢 |
Runs in your editor — drive a whole campaign from the terminal (/potter-run) | 🔴 | 🔴 | 🟢 |
🟡 is a partial version of the same capability; AlphaEvolve's are that capability at the code level; — is a closed service whose behaviour is not documented. GEPA keeps a candidate pool within a run — its 🟡 on knowledge is about what survives it. One documented exception to the pricing row: a DSPy call replayed from DSPy's own cache is not counted.
Where the others are ahead. GEPA has reach — DSPy's adapters, tracing and audience, which is why PromptPotter also installs as a DSPy optimizer. Opik ships six optimizers behind one API; Langfuse has production telemetry. The full argument: related work.
The bench
Peers ship an algorithm; PromptPotter ships an algorithm and the interface around it, and is built to run other people's optimizers inside that interface — one grader, one budget meter, one exam, one stopping rule, so their numbers may sit in one table (what is held equal). Manifests for potter, capo, levi and gepa are in the tree, and each has run end to end in a real campaign; no comparison number is published until the controlled run across all four, scored on one held-out bench set.
What we measure on, each dataset's starting score and the protocol a published number must pass: promptpotter.com/benchmarks.
Extending the engine
git clone https://github.com/PromptPotter/prompt-potter-optimizer && cd prompt-potter-optimizer
python scripts/gate.py # every check CI runs; the first run builds .venv from uv.lock
The optimizer is itself a pipeline of nodes, so it can be pointed at its own optimizer prompts — an outer campaign whose individuals are configurations of the inner optimizer (self-optimization, L4). Every optimizer runs end to end at zero spend (offline-run.md). Start at the developer guide — the engine's four seams on one page — then CONTRIBUTING.md.
In depth
Each cycle is a directory: dashboard.json (live state), rounds/, readout.log (the terminal, mirrored) and log.md, a per-round digest (reading the output). Verbatim, from the round replayed above — what the artifact looks like, not a result to read a number off:
- improved: **yes**
- samples: 34
- composite_fitness: `0.4237`
- ability θ: `+0.308` (ruler 5cd26f22e62708a4, 68 cells, 1PL)
- overlap: 34 shared cells, +16 measured: C0 41.2% → C1.2 44.1% → C4.1 52.9%
- verdict: C4.1 won on θ lift +0.072 (θ +0.293 vs parent +0.221, se 0.196); runner-up C4.2 at +0.005
- cost: **$0.0311** (optimizer $0.0048 c23% ·w4384 · backend $0.0137 c17% · bench $0.0127 c37%)
P(best) trajectory:
f8356ad77a ▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅ 50.0% [winner]
6f7d643a1e ▇▆▅▅▅▅▅▅▅▄▅▄▅▄▅▅▅▅▅▅▅▅▅▅▅▅▅▅▄ 49.7%
a2169d009a ▂▂ 15.8%
A percentage in a round banner is an in-campaign reading, computed on the rows that selected the winner — not a held-out result, and a level should never be compared across rounds. A round is won on θ, not on the accuracy column, and the parent is never re-read on the cells that crowned it, so the lift is not measured against its luck. Which figures are clean, which are biased upward, and what we do about it: benchmarks.md.
- What does L1 actually mutate? The prompt template's fields (persona, task instruction, …) plus whatever your
pipeline.yaml declares as tunable. See the-loop.md. It can draw suggestions from block packs you install — personas, thinking styles and answer formats, each citing the paper it came from — plus what earlier runs turned up, be restricted to them, or run without them.
- My scoring formula was wrong — did I lose results? No. The optimizer rescores on load and replays decisions; on divergence, fork. See
scoring-and-memory.md.
- What if it stalls? Stall and failure are different triggers. Failures route back to the proposing layer (self-healing); stalls escalate L1 → L2 → L3 (the-loop). Stuck for other reasons: troubleshooting.
- Why not search at request time instead? PromptPotter finds a configuration once; from then on it is an ordinary call with no search in the path. Test-time compute, moved out of the request.
- Is this a model router? No. Live model selection per question is a gateway (OpenRouter, OmniRoute). PromptPotter tunes the thing a gateway ends up calling, so the two compose.
- Where are the benchmark numbers? Not published yet, on purpose: DSPy's optimizers and CAPO on BBEH, every method on the same held-out rows, target model and budget held constant. ·
Limitations
- Parameter-based optimization only. It optimizes any pipeline that exposes tunable parameters — prompts, thresholds, model settings — but not model weights, neural architectures, or modality-specific representations.
- Requires a labeled dataset. Input/output pairs are mandatory.
- In-campaign figures are not held-out figures. The winner's own number is read off the rows that selected it, so it overstates deployment performance. The headline is its score on a held-out bench set no optimizer sees:
python -m promptpotter bench.
Documentation
Roadmap · docs index · stable API
Citation
@software{promptpotter,
title = {PromptPotter: LLM-Driven Evolution of Prompts and Pipelines},
author = {Streuli, David},
year = {2026},
url = {https://github.com/PromptPotter/prompt-potter-optimizer}
}
