understudy

understudy: Scenario Testing for AI Agents

Understudy is a scenario-driven testing framework for AI agents that simulates realistic multi-turn users, runs those scenes against an agent through a simple app adapter, records a structured execution trace of messages, tool calls, and handoffs, and then evaluates behavior with deterministic checks, optional LLM judges, and run reports.

How It Works

Testing with understudy is 4 steps:

1. Wrap your agent — Adapt your agent (ADK, LangGraph, LangChain, HTTP) to understudy's interface
2. Mock your tools — Register handlers that return test data instead of calling real services
3. Write scenes — YAML files defining what the simulated user wants and what you expect
4. Run and assert — Execute simulations, check traces, generate reports

The key insight: assert against the trace, not the prose. Don't check what the agent said—check what it did (tool calls).

Two Evaluation Paradigms

Conversational Agent Evaluation

Simulate multi-turn conversations with personas to test dialog agents.

• Use case: Customer service bots, assistants, chatbots
• Assert on tool calls: trace.called("tool_name")

Agentic Flow Evaluation

Evaluate autonomous agents executing multi-step tasks.

• Use case: Code agents, research agents, task automation
• Assert on actions: trace.performed("action")

See examples/README.md for complete examples of both paradigms.

See real examples:

• Example scene — YAML defining a test scenario
• ADK test file — pytest assertions against traces
• LangGraph test file — same tests, different framework
• Agentic test file — agentic flow evaluation
• Agentic scene — task-based scenario
• Example report — HTML report with metrics and transcripts

State injection and next-turn tests

Restore a completed checkpoint, inject synthetic history/state, or replay through exchange X, then execute and assert only against exchange X+1. ADK and LangGraph support isolated checkpoint forks; LangChainApp builds an instrumented LangChain agent with tool middleware. Traces retain nested invocation identities.

See the state-testing guide for run_turn, replay_to, scoped tool mocks, pytest fixtures, CLI commands, and framework-specific restoration limits.

Installation

pip install "understudy[all]"

Quick Start

1. Wrap your agent

from understudy.adk import ADKApp
from my_agent import agent

app = ADKApp(agent=agent)

2. Mock your tools

Your agent has tools that call external services. Mock them for testing:

from understudy.mocks import MockToolkit

mocks = MockToolkit()


@mocks.handle("lookup_order")
def lookup_order(order_id: str) -> dict:
    return {
        "order_id": order_id,
        "items": [{"sku": "BACKPACK-01", "quantity": 1}],
        "status": "delivered",
    }


@mocks.handle("create_return")
def create_return(order_id: str, item_sku: str, reason: str) -> dict:
    return {"return_id": "RET-001", "status": "created"}

3. Write a scene

Create scenes/return_backpack.yaml:

id: return_eligible_backpack
description: Customer wants to return a backpack

starting_prompt: "I'd like to return an item please."
conversation_plan: |
  Goal: Return the hiking backpack from order ORD-10031.
  - Provide order ID when asked
  - Return reason: too small

persona: cooperative
max_turns: 15

expectations:
  required_tools:
    - lookup_order
    - create_return
  forbidden_tools:
    - issue_refund

4. Run simulation

from understudy import Scene, run

scene = Scene.from_file("scenes/return_backpack.yaml")
trace = run(app, scene, mocks=mocks)

assert trace.called("lookup_order")
assert trace.called("create_return")
assert not trace.called("issue_refund")

Or with pytest (define app and mocks fixtures in conftest.py):

pytest test_returns.py -v

Suites and Batch Runs

Run multiple scenes with multiple simulations per scene:

from understudy import Suite, RunStorage

suite = Suite.from_directory("scenes/")
storage = RunStorage()

# Run each scene 3 times and tag for comparison
results = suite.run(
    app,
    mocks=mocks,
    storage=storage,
    n_sims=3,
    tags={"version": "v1"},
)
print(f"{results.pass_count}/{len(results.results)} passed")

Simulation and Evaluation

Understudy separates simulation (generating traces) from evaluation (checking traces). Use together or separately:

Combined (most common)

understudy run \
  --app mymodule:agent_app \
  --scene ./scenes/ \
  --n-sims 3 \
  --junit results.xml

Separate workflows

Generate traces only:

This is the beginning of the README. Read the rest at the original. Read the full README ↗