Agent Graph Builder
Build an agent workflow as a graph, run it, and inspect every node's input and
output. Agent Graph Builder is a Next.js Studio (apps/studio), a
FastAPI + LangGraph execution API
(backend/), and a typed TypeScript SDK published as
@bstockwelldev/agent-graph-sdk (1.1.1 on npm).
Try it
Live: https://agent-graph-builder-app.vercel.app
No sign-up or API key needed. The Run panel defaults to the Stub (offline)
provider, a deterministic stand-in model that needs no key.
- Open Classify & Route (demo) from the graph picker. It opens by itself when
you have no graph, or press
⌘O.
- Press Run in the bottom bar, then Run in the panel. The sample
question ("How does a database index work?") is already filled in.
- Click any node on the canvas, then its Run tab, to see the exact input
and output from that node in the last run. The router shows which edge it picked.
The demo classifies a question as technical or other and then routes it.
Technical questions go to a deterministic tool lookup. Everything else goes to
a free-form LLM answer. Ask something non-technical to see the other branch run.
What the Studio does
- Author graphs visually: React Flow canvas, a node palette, per-type config
forms, typed ports with compatible-target hints, and live validation that
marks problems on nodes and edges. A raw JSON/YAML editor covers everything
the forms don't.
- Run and debug: runs stream node events live. The Studio also has a run
waterfall, run history, and a console drawer, and clicking a diagnostic
jumps to the node or edge it's about.
- Release and replay: publish immutable releases, diff a release against
another release or the unsaved canvas, and replay a past run. You can replay
it frozen, or as a counterfactual (force a different route, or swap an LLM
node's provider or model).
- Agents: an agent is a graph plus an LLM profile (the run's default
model), a system prompt and instructions (prepended to every LLM and tool
loop node), and a tool allow-list (a run that would call anything else is
refused with a blocking diagnostic). Run one from its graph's Run panel
("Run as"), from the Agents panel, or with
/run @agent … in chat. The run
records agent_id, and its graph snapshot records the applied prompts.
- Supporting surfaces: a per-graph knowledge base (RAG), versioned reusable
resources (prompts, tools, LLM profiles, policies), fixture-based simulation,
a chat scratchpad, and usage/spend analytics. Workspace policies and the
resource panels share a Scope picker: the whole workspace, or one graph;
Analytics has its own This graph / Workspace toggle.
- Navigation: the studio is the canvas.
/ reopens the graph you had
open (or shows an empty canvas with the graph picker, also ⌘O);
resources, Analytics, workspace policies and the GenUI catalog are panels,
opened from the graph header's ⋯ menu or ⌘K. The old page URLs
(/prompts, /analytics, /policies, /graphs, …) redirect to them.
Node vocabulary
The backend defines 14 NodeTypes (backend/app/models.py), and the palette
can add all of them:
| Node | What it does |
|---|
input | Defines a run variable (default question) and is the graph's entry point |
prompt | Renders a text template from graph state |
llm | Calls a chat model through the provider-neutral ChatModel protocol |
tool | Runs a tool: a built-in (lookup_topic, web_search, calculator), a registry tool (mock echo), or a tool on an MCP server |
router | Picks exactly one outgoing edge |
branch | Substring gate with its own conditional and default out-edges |
tool_loop | Multi-step tool-calling agent, capped at a configured number of steps |
guardrail | Input-safety checks (length, URLs, injection phrases); fails the run on a violation |
rubric | Static prompt-quality findings; can block the run |
human_gate | Pauses the run for approval. The Run panel shows the checkpoint (message, GenUI surface, reason) with Approve / Reject. The surface can show summaries, charts, tables, diffs, diagrams and inputs, bound to the run's data with {"$ref": "/nodes/<id>/output"} (see /genui); the API is POST /api/runs/{id}/resume with approve, reason and values. On approve, the surface's form values become the run variable named after the gate, so later templates read {gate_id[field]} |
subgraph | Runs another saved graph as a nested run with its own trace |
code_exec | Declares a code-execution contract (validated and passed through; no sandbox yet) |
transform | Deterministic reshape (select / wrap / format message / coerce), inline or from the Transforms library |
output | Returns the run result |
There are three edge kinds: sequence (always), conditional (match upstream
text), and default (the router's fallback).
Model providers
Providers are chosen per run in the Run panel, or with provider on
POST /api/runs. The adapters live in backend/app/providers/.
| Provider | Use it for | Needs |
|---|
| Stub | Demos, CI, offline development | Nothing (Studio default) |
| Groq | Hosted open-weight models | GROQ_API_KEY |
| Google Gemini | Gemini via the Generative Language API | GOOGLE_GENAI_API_KEY or GOOGLE_API_KEY |
| Azure OpenAI | An Azure deployment | AZURE_OPENAI_API_KEY, AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_DEPLOYMENT_NAME |
| Ollama | Local models | Ollama running on the host (default model qwen2.5:3b) |
| OpenAI-compatible | OpenAI, OpenRouter, LM Studio, vLLM | OPENAI_COMPAT_BASE_URL, OPENAI_COMPAT_API_KEY, OPENAI_COMPAT_DEFAULT_MODEL |
Stub classifies with the same keywords as the demo's tool node. Other LLM nodes
get a deterministic [stub answer] … string. The variable names are listed in
backend/env.template.
How it works: the seams
The Studio is not a hard-coded pipeline with a canvas on top. Three seams keep
authoring, execution, and model choice independent:
- Execution follows the graph.
compile_workflow in
backend/app/runtime.py compiles the canonical graph JSON into a real
LangGraph StateGraph: one add_node/add_edge call per node and edge
in the document. There is no if/else chain describing "the" workflow.
Add a node in the editor and the runtime takes a different path.
backend/app/adapters.py wraps this as a RuntimeAdapter with an explicit
capability matrix, so LangGraph is the first compile target, not the only
possible one.
- Models are behind a protocol. LLM nodes call the
ChatModel protocol
(backend/app/providers/base.py), a single async generate(...) method.
Adding a provider means adding an adapter. The compiler and node
executors don't change.
- Execution is observable per node. Every node emits events from a
typed protocol (
backend/app/events.py): node.started,
node.completed, node.failed, node.paused, and edge.selected,
between run.started and run.completed/run.failed/run.paused.
Events stream over SSE (resumable with Last-Event-ID) and are
snapshotted with each run's node traces. The Studio's trace view, the
waterfall, and replay all read from this protocol.
Run it locally
Bare metal
Prerequisites: Node 20+, pnpm 10 (corepack enable picks up the pinned
version), Python 3.11+, and uv.
Backend (terminal 1):
cd backend
uv sync
uv run uvicorn app.main:app --reload --port 8000
Studio (terminal 2, from the repo root):
pnpm install
pnpm run build:sdk
pnpm dev
Open http://localhost:3000. The Studio proxies /api/* to
http://127.0.0.1:8000 (override with API_PROXY_TARGET). The demo graph is
seeded on first backend start, and graphs persist to backend/graphs.db
(SQLite; override with GRAPH_DB_PATH).
Docker
docker compose up --build
This starts the Studio on http://localhost:3000 and the API on
http://localhost:8000, both with hot reload from bind-mounted source. Saved
graphs live in the graph_db volume. docker compose down keeps it, and
down -v drops it. Ollama isn't containerized on purpose: the backend reaches
the host's Ollama at http://host.docker.internal:11434, so it reuses models
you've already pulled.
scripts/dev.ps1 up (PowerShell) and scripts/dev.sh up (bash) wrap the same
compose commands, with -d, down, and down -v.
Keys
Put provider keys in backend/.env.local (gitignored), or point
SHARED_ENV_FILE at a dotenv file. Keys already in the environment are never
overwritten. Without any keys, everything runs on Stub.
Knowledge-base embeddings
Uploaded knowledge documents are embedded by the first configured provider:
| Provider | Model | Needs |
|---|
| Supabase (default) | gte-small, 384-dim, run by the agb-embed Edge Function (supabase/functions/agb-embed); no third-party key | SUPABASE_URL + SUPABASE_SERVICE_ROLE_KEY (SUPABASE_EMBEDDINGS_URL instead of SUPABASE_URL locally, so storage stays on SQLite) |
| OpenAI | text-embedding-3-small (OPENAI_EMBEDDING_MODEL) | OPENAI_API_KEY |
| Google | gemini-embedding-001 (GOOGLE_EMBEDDING_MODEL) | a Gemini key |
EMBEDDING_PROVIDER=supabase|openai|google forces one. A graph stays on the
provider it was first indexed with, because vectors from different models
can't be compared. The Knowledge panel shows the provider, model and vector
size in use. Deploy the function with
supabase functions deploy agb-embed (JWT verification on; it only accepts
the service-role key). Groq has no embedding models, so it isn't an option.
No provider key? A fake embedder lets knowledge uploads and retrieval work
locally (e2e uses the same one):
pnpm dev:embeddings # scripts/fake-embeddings.mjs on :8123
# then start the API with:
SUPABASE_EMBEDDINGS_URL=http://127.0.0.1:8123/embed SUPABASE_SERVICE_ROLE_KEY=fake-embeddings \
uv run uvicorn app.main:app --reload --port 8000
With Docker: scripts/dev.sh up --fake-embeddings (or
.\scripts\dev.ps1 up -FakeEmbeddings), which adds
docker-compose.fake-embeddings.yml. Its vectors are hashed bags of words,
not semantic embeddings, and the Knowledge panel still reports them as
Supabase gte-small. Re-upload documents after switching to a real provider.
Leave SUPABASE_URL unset while using it, or graph storage moves to Supabase.
Tests
# Backend (uses the Stub provider; no keys or Ollama needed)
cd backend
uv sync --extra dev
uv run pytest
Backend tests currently write their fixture graphs into the dev database
(backend/graphs.db). Delete that file afterwards if you want a library that
contains only the demo.
# SDK build + SDK and Studio tests (from the repo root)
pnpm test
pnpm run test:coverage # same, with V8 coverage (text summary + coverage/ JSON)
cd backend && uv run pytest --cov=app # backend coverage
CI reports coverage for the backend, SDK and Studio in each job's summary.
The e2e job adds the Studio code the browser run exercises
(E2E_COVERAGE=1 pnpm run e2e locally; the report is in e2e/coverage/html).
It's report-only: there are no thresholds.
End-to-end tests (e2e/, Playwright). They run the built Studio
against a live backend on the Stub provider, using a throwaway SQLite
database, so no keys are needed. A second backend on :8001 runs with
PUBLIC_DEMO_MODE=1 for the demo-mode specs, and an axe scan checks the
main pages for new serious accessibility violations:
pnpm --filter @bstockwelldev/agent-graph-e2e exec playwright install chromium # once
pnpm run e2e # builds the Studio, starts API :8000 + Studio :3100
E2E_SKIP_BUILD=1 pnpm run e2e # reuse an existing apps/studio/.next build
Locally, servers already listening on :8000 / :3100 are reused, so you can
keep them running while you iterate. Start the APIs with CHAT_PROVIDER=stub
(and PUBLIC_DEMO_MODE=1 on :8001) when you do. On failure, pnpm --filter @bstockwelldev/agent-graph-e2e run report
opens the HTML report with traces.
Backend smoke run of the demo graph, printing the full event stream:
cd backend
CHAT_PROVIDER=stub uv run python smoke_test.py
CI and publishing
GitHub Actions runs uv run pytest in backend/ on push and pull requests
(see .github/workflows/ci.yml). No secrets are required — tests use the
Stub provider.
Planning specs: Locked next-set feature plans live in
docs/planning/ (shell resilience → canvas orientation →
from-scratch authoring). Product roadmap: docs/planning/roadmap.md.
To push this repo to GitHub, add a remote and push:
git remote add origin https://github.com/bstockwelldev/agent-graph-builder.git
git push -u origin master
Vercel (production)
Health check: GET /api/health → {"ok":true,"storage_backend":"supabase"} (prod uses Supabase Storage). The production hostname is kept by the operator, not documented here.
CI (.github/workflows/ci.yml) runs backend pytest, Studio
lint/typecheck/test/build, SDK package checks, and the Playwright e2e suite
on every push and PR. A nightly Production smoke workflow runs e2e/smoke/ against
the deployed Studio. It checks health on shared storage in demo mode and a
Stub run of the seeded demo. Set the repository variable PROD_SMOKE_URL to
enable it. The
SDK is released manually with the Release SDK workflow. See
AGENTS.md for the changeset and OpenAPI-contract steps.
Deploy (Vercel)
Production is https://agent-graph-builder-app.vercel.app. vercel.json
deploys the Studio and the FastAPI backend as two services, with /api/*
routed to the backend. Health check: GET /api/health returns
{"ok":true,"storage_backend":"supabase", ...}.
vercel deploy --prod
Set these in the Vercel dashboard. Never commit them.
| Variable | Required | Purpose |
|---|
PUBLIC_DEMO_MODE | Set in vercel.json | 1 = anonymous public deploy: see Public demo mode below |
SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY | Yes (prod) | Durable storage in Supabase Storage |
SUPABASE_STORAGE_BUCKET | Optional | Private bucket; default agent-graph-builder |
OBJECT_STORE_BUCKET, OBJECT_STORE_ACCESS_KEY_ID, OBJECT_STORE_SECRET_ACCESS_KEY | Alternative | S3-compatible store (R2, S3, MinIO, Azure S3 API) |
OBJECT_STORE_ENDPOINT, OBJECT_STORE_REGION | Optional | Custom endpoint / region (auto for R2) |
TURSO_DATABASE_URL, TURSO_AUTH_TOKEN | Alternative | Turso libsql |
GROQ_API_KEY | Optional | Live LLM runs (CHAT_PROVIDER=stub is the deployed default); unused by visitors while PUBLIC_DEMO_MODE=1 |
Storage selection, first match wins: Supabase, then OBJECT_STORE_*, then
TURSO_*, then file SQLite. On Vercel with no shared store, the API returns
503 rather than write to a per-isolate /tmp database that would lose data.
Legacy aliases still resolve: agent-graph-builder-poc.vercel.app and
theagenticengineer-graph-builder.vercel.app. The bare
agent-graph-builder.vercel.app belongs to a different account.
Public demo mode. The prod Studio needs no login and the backend has no sessions, so vercel.json sets PUBLIC_DEMO_MODE=1. With it on:
-
Only Stub runs on the server. Groq, Google, Azure, OpenAI-compatible and Ollama runs need an API key sent with the request (the Run panel's key field). Without one, POST /api/runs, release runs and chat messages return 403 live_provider_requires_api_key. /api/providers/{p}/ready reports not ready and /credentials reports configured: false, so a server key is never offered.
-
Knowledge uploads return 403, and retrieval is skipped, because embeddings would run on the server's key.
-
Workspace policies are read-only: PUT /api/policies/workspace returns 403 workspace_read_only, and any stored workspace overrides are ignored, so no visitor can make a rule block everyone's runs. Each graph's own Policies panel still works on graphs visitors create.
-
Each visitor IP has a write budget (backend/app/public_writes.py). Creating a stored object costs one unit from the create budget: a graph (including a saved copy), a library resource, a release, a resource version, a dataset or a policy exception. Saving something that already exists costs nothing. Runs and simulations use a separate run budget. Over budget, the request returns 429 write_rate_limited with Retry-After. The counts live in the shared store (one small file per hashed IP under write_quotas/), so every serverless instance sees them. Defaults, overridable per deploy:
| Variable | Default |
|---|
PUBLIC_CREATE_LIMIT_PER_HOUR / _PER_DAY | 30 / 100 |
PUBLIC_RUN_LIMIT_PER_HOUR / _PER_DAY | 120 / 500 |
-
GET /api/health includes "public_demo_mode": true. Check it after each deploy.
To use the server's own keys on a private deploy, remove PUBLIC_DEMO_MODE from vercel.json (or set it to 0 there).
The seeded demo is read-only (every deploy, not only public mode). demo_classify_and_route can't be deleted, and PUT returns 403 graph_read_only unless the body matches the canonical demo apart from layout. Layout-only saves return 200 but aren't stored. The demo's graph policies, policy exceptions and knowledge are locked too. In the Studio, Run on an unedited demo runs it as-is; Save or Run after a real edit saves a copy (graph_<id>, "… (copy)") and opens it. On each cold start the backend re-seeds the demo if it's missing or differs from backend/app/demo_graph.py.
What shipped, and what's still simplified
This started as a thin proof of concept for the Provider-Neutral, Graph-Native
Agent Workstream Platform EDD (agent_orchestration_edd.md). Much of the EDD
has shipped since. The rest is still simplified on purpose, behind the same
seams, so each piece can be swapped without touching the others.
Shipped:
| EDD concept | Where it lives |
|---|
| Full node taxonomy: guardrails, rubric, human approval, tool loop, subgraphs, RAG | models.py, nodes.py, knowledge.py, subgraphs.py |
| Typed ports and contract/policy validation | ports.py, contracts.py, policies.py |
| Immutable, fingerprinted releases and semantic diffs | releases.py, fingerprint.py |
Version-pinned run identity (RunGraphSnapshot) | models.py, runtime.py |
| Replay and counterfactual replay | replay.py |
Durable human_gate checkpoints: resume on any instance or after a restart; full run event history in every backend | runtime.py, storage.py |
| Versioned reusable entities (prompts, tools, LLM profiles) | resource_versions.py |
| Agents: graph + LLM profile + instructions + tool allow-list | agents.py |
| Runtime adapter boundary with a capability matrix | adapters.py |
| Durable storage for graphs, releases, and completed runs | storage.py, supabase_store.py, object_store.py |
| Six provider adapters behind one protocol | providers/ |
| MCP tools (HTTP transport; write-only request headers, tool discovery), telemetry (Langfuse, opt-in), usage analytics | mcp/, telemetry/, analytics.py |
Still simplified:
| Today | EDD target | Swap point |
|---|
Live SSE event buses are in memory (RUN_BUSES). Paused and finished runs, with all their events, persist, and a paused run resumes on any instance (its checkpoint is stored; its API key is not, so a keyed provider needs it resent). But a live stream exists only in the instance running the run; on Vercel runs finish inside the request, and other instances replay events from storage. | Cross-instance live streaming | events.py → an append-only event log |
| No multi-tenancy. There's optional Studio sign-in, but no orgs or projects, no per-tenant isolation, and no RLS. Every graph is visible to every user of a deployment. | Supabase Auth + RLS + org/project hierarchy | storage.py, Studio middleware |
| Storage is a JSON document store (Supabase Storage, S3, Turso, or SQLite), not relational tables. | Supabase PostgreSQL | storage.py |
code_exec declares a contract but has no sandbox executor. | Sandboxed execution | nodes.py |
| No per-node retry policy. | Retries with backoff, visible in the waterfall | runtime.py, nodes.py |
More
