Orchestra Agent Tools

Type /orchestra:onboard in Claude Code — or the matching onboarding command in your agent — and it walks your LLM app from captured traces to a measured, cheaper — often local — model, with evidence you can trust.

Works with Claude Code · Cursor · Codex · Devin (OpenCode and Hermes Agent: deprecated)

---

Orchestra (formerly Understudy) maintains this public, MIT-licensed skill library plus a thin CLI. It gives your coding agent the playbooks to evaluate an AI workload locally, optimize it, and route it to a better-value model — local-first, with no uploads, no provider calls, and nothing spent by default. The CLI and plugin are orchestra (understudy still works as a CLI alias, and rerunning the installer replaces the old understudy plugin).

Get started (30 seconds)

curl -fsSL https://raw.githubusercontent.com/understudylabs/orchestra-agent-tools/main/install.sh | bash

You need macOS or Linux (on Windows, use WSL) and Node.js 22.19 or newer. The installer checks both before it changes anything and tells you the fix for your machine; with Homebrew it offers to install Node for you. Run orchestra doctor any time to see what Orchestra can find. The installer then installs the CLI and hands over to orchestra init, a short guided setup:

1. What to set up: Orchestra account + coding-agent plugin (recommended), the same plus the local model lab (Apple Silicon: the Orchestra 2B model, about 3.6 GB, downloads in the background; its MLX runtime needs uv, installed with Homebrew only if you say yes), choose agents, or just the CLI.
2. A checklist that ticks off the CLI, your sign-in (we email a one-time code you type right there; an existing sign-in is kept) and the plugin for every coding agent it finds: Claude Code, Codex, Cursor, Devin.
3. Where do you want to start? Claude Code (or the Codex CLI) opens here with Orchestra ready. For the Claude desktop app, ChatGPT (Codex) or Cursor, the prompt is copied and the app is opened; paste it into a new chat. Run orchestra start to get back to this screen any time.

Anything that needs your attention is listed once, at the end.

The CLI comes from the published npm package @orchestraai/agent-tools. If npm has no release yet (or with --from-source), the installer builds it from the GitHub main branch instead.

The default install uses the smaller Vercel conversation backend. The optional Pi backend can be added later without reinstalling:

npm install -g @earendil-works/[email protected] @earendil-works/[email protected]

Then, in Claude Code:

/reload-plugins
/orchestra:onboard

That's it. Onboarding profiles your machine, asks what you're working on, and points the plugin at your real workload. Local work needs no account. On other agents the activation command differs slightly — see Pick your platform.

Unless you pick the local model lab, the installer does not download model weights or set up MLX, and it never makes frontier calls or spends money.

Non-interactive installs use option 1 without prompting (sign in later with orchestra login); add --local-models for the lab on Apple Silicon or --skip-login to skip the sign-in prompt:

curl -fsSL https://raw.githubusercontent.com/understudylabs/orchestra-agent-tools/main/install.sh | bash -s -- --yes

Override the agent prompt with --agents auto|all|claude-code|cursor|codex|opencode|hermes|devin|none.

The installer is resumable. It writes step markers under ~/.understudy/agent-tools/install-state; after a failed run:

curl -fsSL https://raw.githubusercontent.com/understudylabs/orchestra-agent-tools/main/install.sh | bash -s -- --resume

Or jump directly to a step:

curl -fsSL https://raw.githubusercontent.com/understudylabs/orchestra-agent-tools/main/install.sh | bash -s -- --from-step 2

Developer install from a clone:

npm install
npm run build
node dist/bin.js --help

Update Orchestra Agent Tools

Rerun the same installer. It installs the newest CLI release and refreshes the selected agent adapters and skills; your sign-in is kept:

curl -fsSL https://raw.githubusercontent.com/understudylabs/orchestra-agent-tools/main/install.sh | bash

The install-agent-adapter skill repairs or refreshes how an agent discovers the existing skills. It does not update the underlying checkout by itself. Platforms may still require a plugin reload, window reload, restart, or new session after an update.

What it does

• Captures evidence from your real LLM traffic — traces, transcripts, or a public benchmark — into a local, redacted evidence base.
• Builds a real eval — harness, environment, metric, validator, and holdout — so improvements are measured, not vibed.
• Optimizes the workload — prompt optimization (GEPA/DSPy), local model labs, distillation, and RLM training, all runnable on your machine.
• Picks a route — compare model sweeps, estimate hosted costs, then ramp a percentage of traffic to a cheaper model while the rest stays on your current provider.
• Ships a conservative claim packet — what changed, how it was measured, and what it saves, stated only as strongly as the evidence supports.

The whole OSS loop runs locally and is skill-led:

flowchart LR
    A["Capture<br/>evidence"] --> B["Attach harness +<br/>environment"]
    B --> C["Confirm metric,<br/>validator, holdout"]
    C --> D["Rerun<br/>baseline"]
    D --> E["Optimize<br/>workload"]
    E --> F["Conservative<br/>claim packet"]

Registration is not required for that loop. Hosted gateway access is available after orchestra login; browser, channel, daemon, and desktop-runtime commands remain outside this public CLI until intentionally extracted.

How it's built

Orchestra Agent Tools is agent-platform-neutral at the skill layer. All six supported coding agents share the same skills/ tree and differ only in a thin adapter: manifest, install path, reload step, and onboarding invocation. The adapter registry lives in docs/agent-platform-adapters.md and is exposed through orchestra platforms.

Spine	Path	Purpose
CLI	src/	Thin TypeScript shortcuts for auth, artifact checks, and durable runs.
Skills	skills/	MVP progressive-disclosure agent playbooks — the product.
Docs	docs/	Public methodology and release-boundary notes.
Platform adapters	.claude-plugin/, .cursor-plugin/, .codex-plugin/, .opencode/, .hermes/, .devin/, .agents/, AGENTS.md	Thin manifests exposing the same skill tree to each coding-agent surface.
Scripts	scripts/	Repo hygiene checks, not product CLI code.

The CLI stays boring on purpose. Workflow judgment belongs in skills; durable shortcuts belong in TypeScript only when the agent needs reliable execution, auth injection, artifact writes, or a safety gate.

The hosted surface this CLI consumes is documented at docs.understudylabs.com — see open-source/agent-tools for how this repo fits the platform and open-source/cli for the command-level CLI reference.

Pick your platform

The installer autodetects these; each can also be set up by hand. The install-agent-adapter skill contains the agent-run install/update/verify flow for every platform.

Agent harness	Default installer behavior	Activation
Claude Code	Autodetected when claude is on PATH; installs the local Claude plugin.	Run /reload-plugins, then /orchestra:onboard.
Cursor	Autodetected when Cursor is present; links this repo into ~/.cursor/plugins/local/orchestra.	Restart Cursor or run Developer: Reload Window, then ask Cursor Agent to use the Understudy onboarding skill.
Codex	Autodetected when codex is on PATH; registers the local Codex marketplace from .agents/plugins/marketplace.json.	Run /plugins, choose orchestra-skills, install or enable orchestra, then start a new thread if needed.
OpenCode (deprecated)	Installed only with --agents opencode; links the shared skills into ~/.config/opencode/skills.	Restart OpenCode or open a new TUI session, then run /orchestra-onboard.
Hermes Agent (deprecated)	Installed only with --agents hermes; registers a stable ~/.understudy/skills symlink in skills.external_dirs.	Run /reload-skills (or start a new hermes session), then /onboard or ask Hermes to use the Orchestra onboarding skill.
Devin	Autodetected when the DEVIN or DEVIN_SESSION_ID env var is set or ~/.devin exists; the installer builds the GitHub source and links the CLI globally.	Ask Devin: Use the Understudy onboarding skill for this project.

Every path is local-only: installing an adapter does not authenticate, upload data, download model weights, or make provider calls.

All adapters expose the same skill tree; there are no platform-specific forks. For manual setup, removal, or adapter internals, see docs/agent-platform-adapters.md. You can also inspect the live registry from the CLI:

orchestra platforms
orchestra platforms --inspect claude-code
orchestra --json platforms

First commands

orchestra spine              # prints the public workflow
orchestra skills --list      # list every installed skill
orchestra skills --search gateway
orchestra platforms          # supported agent adapters
orchestra doctor             # local environment check
orchestra models runtime doctor  # verify the pinned Apple Silicon VLM engine

spine prints the public workflow and points agents at skills/understudy/SKILL.md.

Understudy Desktop is deprecated; the remaining orchestra desktop commands print a deprecation warning and will be removed in a future release. See docs/desktop-deprecation.md.

The skill tree

skills/understudy/SKILL.md is the public entrypoint. It routes to exactly one capability worker per intent, grouped by journey stage; deeper playbooks live in each skill's references/ directory:

Stage	Skills
Setup & first run	install-agent-adapter, compatibility install shims, onboard, ladder (the onboarding "climb")
Understand & capture	understand-workload, export-trace, ingest-traces (incl. the capture-directory profiler), capture-evidence (incl. the public-benchmark on-ramp), design-simulated-environment
Local models	manage-local-models, run-local-model-lab, recursive-language-model (incl. RLM pedagogical training)
Compare & diagnose	compare-model-sweep, compare-trajectories
Plan hosted runs	plan-hosted-run (provider routing + cost estimation)
Optimize	optimize-workload, optimize-agentic-workload (read-only search loops and state-mutating API workflows)
Train locally	curate-trajectories, distill-classifier, local-distillation-lab (incl. the pedagogical arm)
RL handoff	prepare-verifier-handoff (decide → author env → package → hand off)
Gateway & routing	use-understudy-gateway (incl. the frontier-keys decision), ramp-and-verify, check-routing-health (self-service diagnostics)
Writing & papers	deslop, latex-paper-polish

skills/README.md is the authoritative index with per-skill descriptions — keep it in sync when adding skills. See docs/current-functionality.md for the migration ledger.

prepare-verifier-handoff is intentionally handoff-only. It is for workloads that need a future Orchestra verifier/RL-environment release or an external partner path after local validation and prompt optimization are insufficient. Prime Intellect Verifiers is the current preferred referral for that rung.

Optimizer implementation stays upstream. Do not vendor GEPA or add the full private runtime as a dependency. The implementation contract is documented in docs/optimize-workload-contract.md, the evidence-ladder methodology behind it in docs/methodology-framework.md. The TypeScript-to-uv Python bridge pattern is documented in docs/uv-python-bridge.md.

Public benchmark golden path. For a public demo or agent smoke test, use the public-benchmark on-ramp in skills/capture-evidence/references/public-benchmark-path.md. It points agents at public upstream benchmarks such as Zapier AutomationBench and Harvey LAB. Keep benchmark harnesses upstream; use Orchestra for capture, splits, baselines, optimization, and conservative claims.

Going hosted (optional)

Everything above runs without an account. When you want the hosted gateway, the first auth journey is intentionally narrow:

orchestra login --email [email protected]   # emails a one-time code
orchestra login --code 123456             # completes sign-in (non-TTY shells)
orchestra doctor --hosted
orchestra workloads list
orchestra workloads create classify --capture
orchestra gateway probe --provider anthropic --project rehearsal --workload classify
orchestra captures list --project rehearsal --workload classify
orchestra captures list --project rehearsal --workload classify --from 2026-06-07T10:14:00Z --to 2026-06-07T10:16:00Z --json
orchestra captures export --request-ids-file request-ids.txt --project rehearsal --out .understudy/capture-batch --include-payload --yes
orchestra traces export <trace-id> --project rehearsal --out .understudy/trace-exports --include-payload --yes
orchestra traces export --trace-ids-file trace-ids.txt --project rehearsal --out .understudy/trace-exports --include-payload --yes
orchestra traces export --project rehearsal --workload classify --date 2026-08-29 --out .understudy/trace-exports/day --include-payload --yes
orchestra evals build --project rehearsal --workload classify --name classification-day --out .understudy/evals/classification-day --yes
orchestra routes set classify --project rehearsal --model-id glm-5.1 --traffic-pct 10
orchestra routes show classify --project rehearsal
orchestra routes clear classify --project rehearsal

To correlate a database execution with hosted captures when no request ID was saved, use the timestamp search above. It returns indexed request IDs without downloading payloads. See capture time search for timezones, inspecting candidates, and exporting selected matches.

routes set writes control-plane route config: your application keeps calling the normal gateway while a percentage of traffic goes to the selected Orchestra model and the rest remains passthrough/frontier.

login --email uses the Orchestra email-code registration flow. In an interactive terminal it prompts for the code inline; in a non-TTY shell (a coding agent, a script) it sends the code and exits, and login --code completes the pending sign-in — so an agent can drive sign-up as two plain shell commands. It stores the returned sk_* in ~/.understudy/credentials.json with mode 600 and writes a repo-local .understudy/config.json when the platform returns a default project. run injects UNDERSTUDY_API_KEY, UNDERSTUDY_GATEWAY_URL, and the non-secret UNDERSTUDY_ORG_ID when known into the child process; do not copy secrets into repo files or chat output.

doctor --hosted checks credentials, gateway health, projects, keys, models, and workloads without provider calls. gateway probe is an explicit tiny live call and prints request metadata, not the completion text. If you need BYOK for the probe, pass --byok-env ENV_NAME; the CLI reads the key from the environment and never persists or prints it. captures list/get are metadata-first and redacted by default. Full capture export is opt-in, file-only, and requires --include-payload --yes. For a customer-owned batch, put one request id per line in a file and pass --request-ids-file; the CLI retries transient failures, resumes from completed files, and writes failed-request-ids.txt. Redacted batch files use .summary.json, keeping them distinct from full-payload .payload.json files. traces export can resolve explicit trace IDs through the customer trace request-ID endpoint, or download exactly one raw workload day through --workload. Workload mode uses the rolling latest 24 hours by default; --date YYYY-MM-DD selects a completed UTC calendar day. There is no unbounded --all trace scan. Explicit traces write a private trace.json membership manifest, per-request files, and batch failure manifests. Workload days write only a final ordered source/index.jsonl and source/summary.json after success. models list shows public Orchestra model IDs and display names only.

If the coding agent has an approved native email connector, it may complete the email-code prompt by reading the fresh Understudy sign-in email directly. The agent should search only for the current login email, use the code once, and not print the code or retain it in artifacts.

The hosted contracts behind these commands are documented on the docs site: the control-plane API (what workloads, routes, and captures call), the routing and capture semantics, the gateway request headers and response headers, and the CLI command reference for every command in the journey above.

For agents that read loose .claude/skills/ directories but not plugins, orchestra setup remains as the legacy fallback: it copies the onboarding skill into ./.claude/skills/ (--global for ~/.claude/skills/). Either way, then ask the coding agent to convert the current repo to Orchestra or add a thin GEPA/DSPy optimizer. The installed onboarding skill starts by checking orchestra status --json and stops with a clear login instruction if the user is not authenticated.

CLI reference

The TypeScript CLI currently owns the public tools surface. Every command accepts the global --json flag (before or after the subcommand) and emits machine-readable JSON where supported.

spine
platforms
skills
doctor
login
logout
status
projects
keys
models
workloads
captures
gateway
routes
setup
setup-code
run
capture-evidence   (alias: understand)
capture-import
optimize-workload
route-decision
value
experiments
next

setup-code is skill-routed. It does not patch files directly; it tells the coding agent to use skills/onboard/setup-code.md and the matching framework recipe.

gateway is intentionally probe-only: gateway health checks healthz and gateway probe sends one explicit tiny request. It is not a daemon, proxy, browser, chat runtime, or hosted agent surface. Full-runtime command names such as browser, channels, schedule, daemon, agent, and chat are not registered in this public CLI. Use orchestra skills --search <query> to find the relevant capability skill instead.

For GEPA/DSPy work, the CLI stays as the guide and gate surface while Python is used only for small local optimizer environments:

orchestra optimize-workload adapter run --repo . --adapter dspy-gepa \
  --samples samples.json --input-keys question --output-keys answer \
  --model student-model --reflection-model reflection-model \
  --budget-usd <approved-usd> \
  --input-usd-per-million <conservative-input-price> \
  --output-usd-per-million <conservative-output-price> \
  --num-threads 1 --execute

The adapter creates an approval-gated uv run --no-project --python 3.13 runtime with exact dspy==3.4.1, gepa[dspy]==0.1.4, and cloudpickle==3.1.2 packages plus litellm>=1.83,<2. Climb controls, streaming progress, and the holdout check are described in docs/dspy-gepa-hill-climbing.md. Cloudpickle preserves resumable DSPy signature state. That environment, resumable logs, and owner-only optimizer receipts are local runtime state. They are not package infrastructure and must not be committed.

For the exact before/after functionality ledger, see docs/current-functionality.md.

Privacy

No provider calls, uploads, model downloads, secret-value inspection, or hosted jobs run by default. After authentication, the CLI emits bounded product telemetry documented in docs/telemetry.md; disable it with UNDERSTUDY_TELEMETRY=0.

Read the full policies:

• docs/privacy-and-data-boundaries.md
• docs/security.md
• docs/telemetry.md
• docs/oss-release-boundary.md

Contributing

Agent and contributor conventions live in AGENTS.md; the release process in docs/release-checklist.md. Release archives should pass:

npm run check

Do not commit: customer names, domains, prompts, completions, traces, or datasets; private repo paths or internal-only runbooks; API keys, tokens, provider secrets, or local env files; hosted production URLs except documented public defaults.

Do commit: local-only TypeScript CLI code, public agent skills, synthetic templates and docs, and reproducible command outputs that do not contain private payloads.

License

MIT.