harness-eval

An evaluation framework that definitively ranks agentic coding frameworks by having each one build the same product from the same spec, with the same harness and model, in isolated sandboxes — then grading every artifact on a weighted, evidence-based rubric.

First question answered: which of Superpowers, Compound Engineering, Agent Skills, and GSD builds the best implementation of the Symphony Service Specification (an issue-tracker-driven orchestration daemon) when driven by Claude Code + Opus 4.6? The architecture deliberately holds every other variable fixed so the framework is the only thing being measured — and swaps cleanly for phase 2 (other harnesses like OpenCode/Codex, other models like GLM via z.ai).

Status

Built (merged or in review) — each capability was specified under openspec/changes/ before implementation:

• Core harness — candidate registry, orchestration with isolation + budgets + provenance, headless build sessions, two-instrument grading (evidence-based evaluator + blind code-quality judge), telemetry, and results.json + scorecard.md. (setup-harness-eval-framework)
• Results dashboard — cross-run leaderboard, per-run scorecards, trial drill-downs with evidence, and live re-weighting. (add-results-dashboard)
• PRD library + bring-your-own-PRD — a target abstraction spanning several product shapes (symphony-daemon, cli-tool, plus rest-api and web-app adapted from ViBench), an init --target … --spec scaffolder, and freeze/attribution rules. (add-prd-library)
• Pluggable models — a model-profile registry, worker/judge resolution, GLM via z.ai validated end-to-end, a model probe connectivity check, and cross-vendor-judge + cost-basis caveats in results/scorecards. (add-pluggable-models)
• Pluggable providers/harnesses — Docker, E2B, macOS Virtualization, and worktree providers share the same sandbox contract; harness registry entries are validated per implemented driver. See docs/HARNESS-ONBOARDING.md.

In review on branches: the Eval Studio web UI (review + configure on shadcn/ui — this branch). Not yet built: a language-effectiveness evaluation mode. See ROADMAP.md.

How it works

registry (pinned frameworks) ──┐
eval target (PRD + test plan) ─┼─▶ orchestrator ─▶ per-trial sandbox ─▶ headless build session(s)
run config (budgets, weights) ─┘        │                                   │
                                        ▼                                   ▼
                              provenance + telemetry              archived workspace + transcripts
                                        │                                   │
                                        └────────────▶ grading ◀────────────┘
                                          evaluator (test plan, evidence)
                                          + blind judge (code quality)
                                                       │
                                                       ▼
                                    results.json ─▶ scorecard.md + web dashboard

This is the beginning of the README. Read the rest at the original. Read the full README ↗