Agent Eval Harness

Agent Eval Harness evaluates skills and agent capabilities with one declarative eval.yaml: analyze → generate cases → run → judge → trace in MLflow → optimize. Same config on your laptop, Harbor containers, or EvalHub.

Why Agent Eval Harness

• One config everywhere. eval.yaml drives local runs, Harbor, and EvalHub.
• Skill or prompt mode. Test packaged skills, or agent capabilities directly (including agentic documentation checks).
• Robust scoring. LLM + code judges, pairwise A/B, thresholds, and HTML reports.
• MLflow-native traces. Opt-in experiments, datasets, and hierarchical GenAI traces.
• Close the loop. /eval-optimize proposes skill fixes from failures and re-runs.

Execution Model

The harness separates how many invocations (execution.mode) from what to execute (execution.skill or execution.prompt):

Execution Mode

• case: One invocation per test case (default). The harness loops over cases.
• batch: One invocation for all cases via batch.yaml. The skill/agent loops internally.

What to Execute

• Skill mode (execution.skill): Test predefined skill implementations (/my-skill --args). Evaluates skill correctness, quality, and cost efficiency.
• Prompt mode (execution.prompt) ✨ NEW: Test agent capabilities directly by sending prompts without a skill wrapper. Extensible to any agent evaluation scenario.

Implemented flavor - Agentic Documentation Testing (see examples/openshift-agentic-docs.md):

• Documentation effectiveness: Can agents navigate and use your docs?
• Pattern understanding: Can agents identify and apply code patterns?
• Constraint compliance: Do agents respect documented rules?
• API usage: Can agents correctly use APIs from documentation alone?

Extensible to other scenarios:

• Code generation from specifications
• API usage pattern validation
• Reasoning trace quality assessment
• Custom agent capability benchmarks

Useful for testing documentation quality (CLAUDE.md, AGENTS.md, ai-docs/), onboarding effectiveness, and establishing agent baseline capabilities.

Quick Start

1. Add to your project

Install from the skills registry:

claude plugin install agent-eval-harness@opendatahub-skills

Or clone and load as a local plugin:

git clone https://github.com/fullsend-ai/agent-eval-harness
pip install -e ./agent-eval-harness
claude --plugin-dir ./agent-eval-harness

This makes all eval skills available: /eval-setup, /eval-analyze, /eval-dataset, /eval-run, /eval-review, /eval-mlflow, /eval-optimize, /eval-compare, /eval-anova, and /eval-check.

2. Set up environment

/eval-setup

This checks dependencies, configures MLflow, verifies API keys, and creates directories.

3a. Analyze your skill (skill mode)

/eval-analyze --skill my-skill

This examines the skill's SKILL.md, discovers test cases, and generates eval.yaml with:

• execution.mode: case or batch
• Natural language schema descriptions of your dataset and outputs
• Suggested judges (inline checks + LLM quality assessment)
• Regression thresholds

3b. Analyze for prompt mode evaluation

/eval-analyze --prompt examples/openshift-agentic-docs.md

This analyzes your repository's documentation (CLAUDE.md, AGENTS.md, ai-docs/) and generates eval.yaml with:

• execution.prompt: "{{ input.prompt }}" (prompt mode)
• A generation: block with builtin documentation prompts (docs/navigation, docs/anti-pattern, etc.)
• LLM rubric judges for semantic evaluation
• Documentation tracking to verify agents use docs correctly

Note: Prompt mode is extensible. The OpenShift analysis prompt is a domain-specific example. You can create custom analysis prompts for other domains or agent capability testing scenarios.

4. Generate test cases (if needed)

/eval-dataset

Creates 5 starter test cases based on the skill analysis. Skip this if you already have cases.

5. Run evaluation

/eval-run --model opus

This prepares a workspace, runs the skill (headless or interactive), collects artifacts, scores with judges, and reports results. To run the same eval against an open model through OpenRouter, export OPENROUTER_API_KEY and pass --model openrouter:/<author>/<slug> (see the OpenRouter guide).

eval.yaml

The harness uses natural language to describe evaluation datasets and skills input/output and spawns LLM sub-agents to interpret them.

name: my-skill-eval
description: Evaluate the main skill pipeline

# Execution — what to run and how (runner-agnostic)
execution:
  mode: case              # case (per-case invocation) or batch (single invocation)
  skill: my-skill-name    # skill mode; use `prompt:` instead for prompt mode
                          # (direct agent invocation, no skill wrapper)
  arguments: "{prompt}"   # resolved per case from input.yaml fields
  # timeout: 3600            # Wall-clock timeout in seconds per invocation
  # max_budget_usd: 5.0      # Cost cap in USD per invocation
  # parallelism: 3            # Run up to N cases concurrently (case mode only)
  # env:                     # Inject env vars into workspace settings
  #   JIRA_SERVER: http://localhost:8080   # Literal value
  #   JIRA_TOKEN: $JIRA_TOKEN              # $VAR resolved from caller's env
  # project_hooks: true      # Carry the project's .claude/settings.json hooks
                             # into each workspace (default); false = harness hooks only
  # symlinks: [types]        # Extra top-level project entries to link into each workspace

# Runner — agent harness + runner-specific knobs
runner:
  type: claude-code          # claude-code | cursor | codex | fullsend | cli | responses-api
  # effort: high              # Claude: low..max; Codex: minimal..xhigh
  # settings: {}              # Arbitrary Claude Code settings merged into workspace
  # plugin_dirs: []           # Directories to load plugins from
  # env:                       # Extra env vars for subprocess ($VAR resolves from caller)
  #   CUSTOM_AUTH_TOKEN: "$CUSTOM_AUTH_TOKEN"
  # system_prompt: |          # Appended to Claude CLI system prompt
  #   Custom instructions for the skill run.

# Models — defaults for each role (CLI flags override)
models:
  skill: claude-opus-4-6
  judge: claude-opus-4-6
  # hook: claude-sonnet-4-6  # Model for LLM-based AskUserQuestion answering

# MLflow logging target (optional)
mlflow:
  experiment: my-skill-eval

# Permissions — tool access during headless execution
permissions:
  allow: []            # Tool patterns to allow (empty = all)
  deny:
    - "mcp__*"         # Block MCP tools during eval

# Dataset — where test cases live and what they look like
dataset:
  path: eval/dataset/cases
  schema: |
    Each case directory contains:
    - input.yaml: YAML file. The 'prompt' field is the main input to
      the skill. Optionally 'context' with additional context.
    - reference.md: Gold standard output for comparison scoring.

# Inputs — tool interception for headless/interactive execution
# AskUserQuestion uses 3-tier answering: exact case_overrides →
# LLM call (models.hook) with input.yaml + answers.yaml context → fallback
inputs:
  tools: []
  # - match: Questions asked to the user via AskUserQuestion.
  #   prompt: |
  #     Answer based on test case context in input.yaml and answers.yaml.
  #     Default to "yes" for confirmations.
  # - match: |
  #     Any interaction with Jira — MCP tools or scripts.
  #   prompt: |
  #     Block production Jira. Only allow test instances.

# Outputs — what the skill produces (files on disk or tool calls)
outputs:
  # File artifacts on disk
  - path: artifacts
    # batch_pattern: "RFE-{n:03d}"  # Map output files to cases in batch mode
    schema: |
      One markdown file per case, named NNN-slug.md where NNN is the
      case number (001, 002, ...).

  # Tool call outputs (for side effects like API calls)
  # - tool: mcp__atlassian__create_issue
  #   schema: |
  #     Creates a Jira issue with title, description, priority.

# Traces — execution data to capture for judges
traces:
  stdout: true     # Capture stdout.log
  stderr: true     # Capture stderr.log
  events: true     # Execution events: tool calls, reasoning, results (default: true)
  metrics: true    # Capture exit code, tokens, cost, duration

# Judges — evaluate output quality
judges:
  # Inline code check
  - name: has_content
    description: |
      Check that the generated output is non-empty and has at least
      100 characters of content.
    check: |
      content = outputs["main_content"]
      if len(content.strip()) < 100:
          return False, f"Output too short ({len(content.strip())} chars)"
      return True, f"Output has {len(content.strip())} chars"

  # LLM judge with inline prompt (conditional — skipped when condition is false)
  - name: completeness
    if: "not annotations.get('skip_completeness', False)"  # Skip based on annotations
    feedback_type: int
    score_range: [1, 5]      # declare the scale — omitting it warns at config load
    description: |
      How completely the output covers the reference — one criterion;
      clarity and accuracy belong to other judges.
    prompt: |
      How completely does the generated output cover the reference?

      {{ outputs }}

      Score 1-5: 1 = most requirements missing, 3 = covers the basics
      with gaps, 5 = every requirement addressed.

  # LLM judge with prompt file and supplementary context
  # - name: detailed_quality
  #   description: Detailed quality assessment with rubric
  #   prompt_file: eval/prompts/quality-judge.md
  #   context:
  #     - eval/prompts/scoring-rubric.md
  #     - eval/prompts/domain-guidelines.md

  # External code judge (for complex validation)
  # - name: schema_valid
  #   description: Validate output schema
  #   module: eval.judges.schema_checks
  #   function: check_schema

  # Execution efficiency check (uses trace metrics)
  # - name: cost_reasonable
  #   description: Verify cost stays under $0.50 per case
  #   check: |
  #     cost = outputs.get("cost_usd", 0)
  #     if cost and cost > 0.50:
  #         return False, f"Cost ${cost:.2f} exceeds limit"
  #     return True, f"Cost ${cost:.2f}"

  # Tool call check (uses tool outputs)
  # - name: jira_created
  #   description: Verify the skill created a Jira issue
  #   check: |
  #     calls = outputs.get("tool_calls", [])
  #     jira = [c for c in calls if "create_issue" in c.get("name","")]
  #     if not jira:
  #         return False, "No Jira issue created"
  #     return True, "Created issue"

  # Pairwise comparison judge
  # - name: pairwise
  #   description: Compare two runs and pick the better output
  #   prompt_file: eval/prompts/comparison-judge.md
  #   # model: <model-id>   # Optional override; default is models.judge

# Thresholds for regression detection
thresholds:
  completeness:
    min_mean: 3.5            # Minimum average score
  # has_content:
  #   min_pass_rate: 1.0     # Minimum fraction of cases passing (0.0–1.0)
  # pairwise:
  #   min_win_rate: 0.6      # Minimum pairwise win rate

Key concepts

• execution — mode determines how evaluation runs:

  • case (default, skill mode): Skill invoked once per test case with {field} placeholders resolved from each case's input.yaml
  • batch (skill mode): All cases bundled into batch.yaml for a single skill invocation
  • prompt (prompt mode): Agent receives prompts directly without a skill wrapper, useful for testing agent capabilities like documentation navigation, pattern understanding, constraint compliance, etc.

  Additional fields: arguments template, optional timeout (wall-clock seconds per invocation), max_budget_usd (cost cap per invocation), parallelism (run up to N cases concurrently in case/prompt modes), env for injecting environment variables into workspaces ($VAR syntax resolves from caller's environment), and project_hooks (default true): the project's .claude/settings.json hooks are appended to each workspace's settings after the harness's own SubagentStop/PreToolUse hooks, so a skill keeps the hooks it has in production (a SessionStart/compact recovery banner, a Stop guard). Hook commands run with the workspace as cwd, so a relative command must name a file the harness links into the workspace (scripts/, .claude/, …); set project_hooks: false to run with the harness hooks only. Local workspaces only — a Harbor task package carries eval.yaml permissions, not the project's settings file. symlinks (default []) lists extra top-level project entries linked into each local workspace after the defaults (scripts, .claude, CLAUDE.md, .context, skills) — single names, not names the harness writes into a workspace (rejected at load); a missing entry or one resolving outside the project root and runner.plugin_dirs is skipped with a warning, and one whose workspace path already exists fails staging. Use dataset.workspace.files {dest, source} instead for staging that must also work under Harbor or EvalHub.

• schema — natural language description of structure. Used on dataset and each entry. Agents and judges read these to understand the data.

Example: eval.yaml for RFE Creator

name: rfe-creator
execution:
  mode: batch
  skill: rfe.speedrun
  arguments: "--input batch.yaml --headless --dry-run"
runner:
  type: claude-code
models:
  skill: claude-opus-4-6
  judge: claude-opus-4-6
mlflow:
  experiment: rfe-eval
permissions:
  deny: ["mcp__atlassian__*"]  # Block Jira writes during eval

dataset:
  path: eval/dataset/cases
  schema: |
    Each case directory contains:
    - input.yaml: YAML file. The 'prompt' field is the problem statement
      to send to the skill. 'clarifying_context' has additional context.
    - reference-rfe.md: Gold standard RFE (markdown with YAML frontmatter:
      rfe_id, title, priority, size, status).
    - reference-review.md: Gold standard review (markdown with YAML
      frontmatter: score 0-10, pass bool, recommendation, feasibility,
      per-criterion scores: what, why, open_to_how, not_a_task,
      right_sized each 0-2).
    - annotations.yaml: Expected scores and test metadata.

inputs:
  tools:
    - match: Questions asked to the user via AskUserQuestion.
      prompt: |
        Answer based on the test case. If asked about priority,
        say "Normal". If asked to confirm, say "yes".
    - match: |
        Any interaction with Jira — via MCP tools (mcp__atlassian__*)
        or scripts that import jira-python or call the Jira REST API.
      prompt: |
        Block production Jira. Only allow if JIRA_SERVER points to
        a test instance or jira-emulator.

outputs:
  - path: artifacts/rfe-tasks
    schema: |
      One markdown file per case, named RFE-NNN-slug.md where NNN is
      the case number (001, 002, ...). Contains YAML frontmatter with
      rfe_id, title, priority, size, status.
      Skip files ending in -comments.md or -removed-context.md.
  - path: artifacts/rfe-reviews
    schema: |
      One review file per case, named RFE-NNN-slug-review.md. Contains
      YAML frontmatter with score, pass, recommendation, feasibility,
      and per-criterion scores.

traces:
  metrics: true

judges:
  - name: frontmatter_valid
    description: |
      Validate that each generated RFE has valid YAML frontmatter with
      required fields: rfe_id, title, priority, status.
    check: |
      import yaml
      task = outputs["rfe-tasks_content"]
      if not task.startswith("---"):
          return False, "No YAML frontmatter"
      fm = yaml.safe_load(task.split("---", 2)[1])
      required = ["rfe_id", "title", "priority", "status"]
      missing = [f for f in required if f not in fm]
      if missing:
          return False, f"Missing: {', '.join(missing)}"
      return True, "All required fields present"

  - name: quality
    description: |
      Evaluate quality of the generated RFE compared to the reference.
    feedback_type: int
    score_range: [1, 5]
    prompt_file: eval/prompts/quality-judge.md
    context:
      - eval/prompts/rfe-scoring-rubric.md

  - name: cost_efficient
    description: Verify the pipeline doesn't exceed $1 per case.
    check: |
      cost = outputs.get("cost_usd", 0)
      if cost and cost > 1.0:
          return False, f"Cost ${cost:.2f} exceeds $1.00"
      return True, f"Cost ${cost:.2f}"

thresholds:
  frontmatter_valid: {min_pass_rate: 1.0}
  quality: {min_mean: 3.5}

Example: eval.yaml for Architecture Context

name: architecture-context
execution:
  skill: repo-to-architecture-summary
runner:
  type: claude-code

dataset:
  path: eval/dataset/cases
  schema: |
    Each case directory contains:
    - input.yaml: YAML file. 'repo_path' is the local path to the
      repository to analyze. 'distribution' (rhoai or odh) and
      'version' identify the platform.
    - reference-architecture.md: Gold standard architecture document
      with sections: Architecture Components, APIs, Dependencies,
      Network Architecture, Security. Claims have source references
      in file:line format.

inputs:
  tools:
    - match: Questions asked to the user via AskUserQuestion.
      prompt: |
        If asked which distribution, answer "rhoai".
        If asked which version, answer the latest.

outputs:
  - path: output
    schema: |
      A single GENERATED_ARCHITECTURE.md file per case with markdown
      sections matching the reference structure.

traces:
  metrics: true
  events: true   # Capture tool calls for source reference analysis

judges:
  - name: required_sections
    description: |
      Check that the generated architecture document contains all
      required sections.
    check: |
      content = outputs["output_content"]
      required = ["Architecture Components", "APIs", "Dependencies",
                  "Network Architecture", "Security"]
      missing = [s for s in required if s.lower() not in content.lower()]
      if missing:
          return False, f"Missing sections: {', '.join(missing)}"
      return True, f"All {len(required)} sections present"

  - name: accuracy
    description: |
      Compare the generated architecture summary against the reference.
    feedback_type: int
    score_range: [1, 5]
    prompt: |
      Compare the generated architecture summary against the reference.
      Are the same components identified? Are APIs correct?
      Are dependencies and security details accurate? Score 1-5.

thresholds:
  required_sections: {min_pass_rate: 1.0}
  accuracy: {min_mean: 3.5}

Example: eval.yaml for Prompt-Based Documentation Testing

name: docs-navigation-eval
description: Test if agents can navigate and use repository documentation
# Prompt mode — sends prompts directly to the agent (no skill wrapper)

execution:
  mode: case
  prompt: "{{ input.prompt }}"  # Resolved from input.yaml per case

runner:
  type: claude-code

models:
  skill: claude-sonnet-4-6
  judge: claude-opus-4-6

dataset:
  path: eval/dataset/cases
  schema: "input.yaml with 'prompt' (question) and 'expected_files' (docs to consult)"

# Synthetic generation: a top-level block, peer of execution/dataset/judges
generation:
  strategy: synthetic
  # Repository knowledge injected into every generation prompt
  context:
    documentation_structure:
      entry_point: CLAUDE.md
      areas:
        - path: ai-docs/
          topics: [component-docs, workflows]
    constraints:
      - rule: "All new APIs must start with v1alpha1"
        documentation: ai-docs/practices/api-evolution.md
      - rule: "Never modify files in vendor/"
        documentation: CLAUDE.md
  # Each seed picks a generation prompt via builtin / prompt_file / prompt
  seeds:
    - category: navigation
      builtin: docs/navigation          # from agent_eval/prompts/ (see list_prompts.py)
      count: 10
      description: Finding specific documentation
    - category: anti-pattern
      builtin: docs/anti-pattern
      count: 5
      description: Rejecting constraint violations

outputs:
  - path: outputs
    schema: "agent responses (markdown files)"

traces:
  stdout: true
  events: true
  metrics: true

judges:
  # Check if agent read the expected documentation
  - name: consulted_docs
    builtin: consulted_docs
    if: "annotations.get('category') == 'navigation'"
    arguments:
      min_coverage: 0.8
      match: suffix
  
  # Semantic quality assessment
  - name: answer_quality
    feedback_type: int
    score_range: [1, 5]
    llm_rubric: |
      Evaluate the agent's answer against the expected behavior.
      Score 1-5 where 5 is excellent.

thresholds:
  consulted_docs: {min_pass_rate: 0.8}
  answer_quality: {min_mean: 3.5}

Skills

/eval-setup

Set up the evaluation environment: verify dependencies, configure MLflow tracking and tracing, check API keys, create directory structure.

/eval-analyze

Analyze a target and generate eval.yaml. Two modes:

Skill mode (--skill): Examines the skill's SKILL.md, discovers test cases, and produces configuration with:

• execution.mode: case or batch
• Dataset schema, output descriptions
• Suggested judges (inline checks + LLM prompts)

Prompt mode (--prompt): Uses an analysis prompt to generate evaluation config. Domain-specific analysis prompts (see examples/) analyze repository documentation (CLAUDE.md, AGENTS.md, ai-docs/) and produce:

• execution.prompt: "{{ input.prompt }}" (direct agent invocation)
• A generation: block with seeds (navigation, anti-pattern, authoring, component-usage, architecture)
• LLM rubric judges for semantic evaluation
• generation.context for test generation

/eval-analyze --skill my-skill                           # Skill mode: analyze skill implementation
/eval-analyze --skill my-skill --update                  # Update existing skill mode eval.yaml
/eval-analyze --prompt examples/openshift-agentic-docs.md # Prompt mode: analyze agentic documentation
/eval-analyze --prompt custom.md                         # Prompt mode: use custom analysis prompt
/eval-analyze --assess                                   # Batch: assess which skills would benefit from evals

Prompt mode is extensible. Create custom analysis prompts for other agent capability testing scenarios (code generation, API usage patterns, reasoning quality, etc.).

Batch assessment (--assess): profiles every skill in the project and classifies each as RECOMMENDED / OPTIONAL / SKIP / EXISTS, so you can decide where evals are worth building before analyzing any single skill. It ignores --skill and writes no config.

/eval-dataset

Generate evaluation test cases. Case provenance is set in the config via generation.strategy:

• skill (default — no generation block needed): the agent authors realistic inputs from the skill analysis.
• synthetic: an LLM generates cases from generation.seeds + generation.context (from /eval-analyze --prompt).
• from-traces: cases are extracted from MLflow production traces.

Whether a run creates a fresh set or augments an existing one is derived from the current dataset (empty → fresh; populated → gap-fill) — there is no --strategy flag.

/eval-dataset                              # generate per the config's generation.strategy
/eval-dataset --count 20                   # target 20 cases (skill/from-traces; synthetic uses per-seed count)
/eval-dataset --run-id <id>                # augment, targeting failures from a prior eval run

Synthetic generation uses builtin generation prompts (docs/navigation, docs/anti-pattern, docs/authoring, docs/component-usage, docs/architecture — from agent_eval/prompts/) combined with repository-specific generation.context to create targeted test cases. Extensible with project-specific prompts via prompt_file: or inline prompt:.

/eval-run

Execute the evaluation suite: prepare workspace, run the skill headlessly, collect artifacts, score with judges, detect regressions, and report results.

/eval-run --model opus                          # Run all cases
/eval-run --model opus --parallelism 3          # Run 3 cases concurrently
/eval-run --model opus --cases case-001         # Run specific case
/eval-run --model opus --baseline prev-run-id   # Compare against baseline
/eval-run --model opus --no-llm-judges          # Skip LLM judges

/eval-compare

Compare evaluation results across multiple models or runs. Scans a directory of eval run artifacts (summary.yaml, run_result.json, report.html) and produces a self-contained tabbed HTML comparison report with model cards, quality/cost tables, per-case breakdowns, embedded per-run reports, and LLM-written analysis (Bottom Line, Where Each Model Shined, Shared Weaknesses, Recommendations).

/eval-compare <input-dir>                          # Discover runs and generate the report
/eval-compare <input-dir> --output <dir>           # Custom output directory
/eval-compare <input-dir> --title "Opus vs Sonnet" # Custom report title
/eval-compare <input-dir> --overview "<context>"   # Add a context paragraph

When an /eval-anova anova.json is present in the input directory, the report also gains an ANOVA/Pareto Statistical Significance section. eval-compare works standalone without it (it never imports the stats libraries).

/eval-anova

Design-of-Experiments over a matrix: of agent configurations. eval-run runs one condition; /eval-anova reads the matrix and fans out /eval-run per cell (condition × replication) into standard runs, then computes repeated-measures / mixed-effects ANOVA + a cost/quality Pareto frontier over their summary.yaml files (anova.json) and renders the comparison via /eval-compare. Because the statistics read standard runs, it can also analyze runs produced by a CI fan-out. Requires the anova extra.

/eval-anova --config eval.yaml                # run every cell → analyze → report
/eval-anova --config eval.yaml --dry-run      # design + cost estimate, no execution
/eval-anova --config eval.yaml --analyze-only # re-analyze existing runs + re-render

See eval/anova-example/ for a self-contained worked example (committed sample runs let you reproduce the analysis + report offline).

/eval-review

Interactive human review of eval results. Presents judge scores and outputs, collects qualitative feedback, analyzes patterns, and proposes SKILL.md changes.

/eval-review --run-id 2026-04-04-opus      # Review a completed run
/eval-review --run-id <id> --cases case-003 # Review specific cases

/eval-mlflow

MLflow integration: sync datasets, log run results, attach judge feedback to traces. The agent reads the schema descriptions to understand case structure — no hardcoded field mappings.

/eval-mlflow --action sync-dataset              # Push cases to MLflow dataset
/eval-mlflow --run-id <id> --action log-results # Log scoring results
/eval-mlflow --run-id <id> --action push-feedback # Push judge+human feedback to traces
/eval-mlflow --run-id <id> --action pull-feedback # Pull MLflow UI annotations
/eval-mlflow --run-id <id>                      # Do everything

/eval-optimize

Automated refinement loop: run eval, identify failures, read traces + judge rationale, edit the skill to fix issues, re-run to verify, check for regressions.

/eval-optimize --model opus --max-iterations 3

/eval-check

Scan the full configuration (skills, commands, CLAUDE.md, hooks) as a system. Finds content overlap, trigger collisions, CLAUDE.md duplication, type misclassification, and broken cross-component references. Produces an informational report with restructuring suggestions.

/eval-check                        # Scan and report to harness-report.md
/eval-check --include-global       # Also scan ~/.claude/CLAUDE.md
/eval-check --output my-report.md  # Custom output path

Architecture

agent_eval/              # Python package (config, runner, state)
  config.py              # EvalConfig from eval.yaml
  state.py               # Shared state persistence
  agent/
    base.py              # EvalRunner ABC + RunResult
    claude_code.py       # Claude Code CLI runner
    stream_capture.py    # Stream-json processing + SubagentStop hook
  mlflow/
    experiment.py        # MLflow experiment setup
    trace_builder.py     # Hierarchical trace builder
  cli/
    trace_run.py         # claude-trace CLI

skills/
  eval-setup/            # Environment setup
  eval-analyze/          # Skill analysis + config generation
  eval-dataset/          # Test case generation
  eval-run/              # Evaluation execution
  eval-review/           # Interactive human review
  eval-mlflow/           # MLflow integration
  eval-optimize/         # Automated refinement loop
  eval-compare/          # Cross-run / cross-model comparison report (+ ANOVA stats section)
  eval-anova/            # DoE/ANOVA matrix experiments (orchestrates eval-run)
  eval-check/            # Full-harness configuration health check

Agent Support

The harness is agent-agnostic via the EvalRunner abstraction. Set runner.type in eval.yaml:

runner:
  type: claude-code    # default — uses claude --print

runner:
  type: cli            # opaque CLI runner — delegates to an arbitrary command
  command: "my-runner run {agent} --model {model} --workspace {workspace}"

The cli runner executes a configurable command template with placeholder substitution. See docs/opaque-cli-runner-contract.md for the full contract (placeholders, metrics.json format, what the command MUST and SHOULD do).

Add new runners by subclassing EvalRunner in agent_eval/agent/ and registering in RUNNERS.

MLflow Tracing

The same tracing used by /eval-mlflow is available for standalone skill runs via claude-trace — a drop-in replacement for claude --print that captures stream-json output and builds hierarchical MLflow traces. See TRACING.md for full documentation.

# Install with MLflow support
pip install -e "./agent-eval-harness[mlflow]"

# Run any skill with tracing
echo "/rfe.speedrun --input batch.yaml --headless" | claude-trace --model opus

Dependencies

• pyyaml >= 6.0
• Optional: mlflow[genai] >= 3.5 (for /eval-mlflow and claude-trace)
• Optional: anthropic >= 0.77 (for LLM judges, pairwise comparison, synthetic dataset generation, and hook answering)