Similar pass rate. 17.5x cost differences.
We ran the same Kimi K3 model through nine coding-agent harnesses—12 configurations in total—on the same 30 software-engineering tasks. With the model, tasks, and runtime held constant, changing the harness changed pass rate, cost, cache behavior, and speed.
Full results
| Harness (configuration) | Pass rate | Median cost per pass | Cache, median cell | Median time |
|---|
| Codex | 66.7% | $3.47 | 88.0% | 6m 43s |
| DSH Creator | 63.3% | $3.28 | 84.3% | 6m 44s |
| Claude Code | 63.3% | $18.34 | 67.8% | 9m 38s |
| Pi | 60.0% | $2.43 | 79.4% | 7m 33s |
| DSH PTC | 60.0% | $4.58 | 87.2% | 7m 44s |
| DSH Standard | 60.0% | $3.46 | 86.5% | 6m 17s |
| Oh My Pi | 56.7% | $4.75 | 82.2% | 6m 46s |
| Kimi Code | 56.7% | $3.65 | 88.0% | 7m 56s |
| DSH Minimal | 56.7% | $4.72 | 84.6% | 5m 41s |
| Exo Harness | 53.3% | $1.05 | 70.3% | 6m 17s |
| OpenCode | 50.0% | $3.24 | 78.4% | 6m 27s |
| Hermes | 50.0% | $2.90 | 85.9% | 6m 58s |
The interactive report includes failed runs, total cost per task, cache behavior, speed, and task-level results. For the evaluation design and analysis, read the launch article.
What is in this repository
.
├── benchmark.json # Public benchmark definition
├── cli/index.mjs # `npx @frontierharness/eval`: workspace + skill installer
├── metadata/
│ ├── difficulty.json # Difficulty assignments and source methodology
│ └── harness-versions.json # Harness versions used for the run
├── results/
│ └── eval-data.json # Normalized aggregate and task-level results
├── tasks/<task>/
│ ├── instruction.md # Prompt shown to every harness
│ └── task.toml # Public task metadata and environment definition
└── skills/frontierharness-eval/ # Agent-neutral skill, usable by hand
├── SKILL.md # Evaluation workflow for a third-party harness
├── PROMPT.md # Copy-paste prompt that points an agent at the skill
├── reference.md # Command reference, runner templates, troubleshooting
└── scripts/ # Provisioning, trial runner, scoring, chart, report
The repository intentionally contains results, task definitions, and the evaluation workflow. Internal infrastructure, credentials, runtime identifiers, private evidence, solutions, and deployment configuration are not included.
Evaluate your own harness
This repository ships a reproduction workflow for evaluating another harness on the published task set. It records environment differences from the original baseline run; a matched control run is needed to establish comparability.
Trial network access. Agents have access to the package registries and source hosts needed by the verifiers, including Ubuntu/Debian apt mirrors, PyPI, npm, PyTorch, GitHub download hosts, and Harbor's task registry. Runta applies the allowlist to the whole runtime, including agent containers; any additional Harbor/Pier isolation still applies. Verifiers install dependencies during trials, so allowing only the model provider and uv downloads blocks valid verification. Task images are pulled before the trial policy is applied. The exact hosts are defined in providers.sh.
Each run.json records egress_policy with its mode, scope, and exact allowed_hosts; the same policy is used for both suites and is included in candidate data and reports. The published baselines did not record their applied allowlist, so new runs default to methodology_comparable: false and receive no leaderboard rank. Evaluate a control harness under the same policy and environment before claiming comparability. Resuming with a changed or unrecorded policy is refused; use a new run id for the new policy and preserve earlier evidence.
skills/frontierharness-eval/ is an agent-neutral skill: point any coding agent that reads SKILL.md at it and it will drive the whole run — freezing the golden checkpoint, running every task from an identical fresh restore, scoring the trials, and building the report.
How to use the skill
Let an agent drive it
Install the evaluation skill and its Runta companion skills
with the Skills CLI (Node.js and Git required).
Run these in the project where you use your coding agent:
npx skills add https://runta.com/docs --skill runta-installer runta-cli
npx skills add frontier-harness-eval/eval --skill frontierharness-eval
runta-installer handles Runta tooling setup and verification; runta-cli provides
runtime operation guidance. frontierharness-eval drives the benchmark, scoring, and
report. If the Runta skills are already installed, only the second command is needed.
Choose the same agent for both commands when prompted, or select one directly:
npx skills add https://runta.com/docs --skill runta-installer runta-cli --agent codex -y
npx skills add frontier-harness-eval/eval --skill frontierharness-eval --agent codex -y
Add --global to both commands to make the skills available across projects. Start a new agent session
in the project and ask:
Use the frontierharness-eval skill to evaluate https://github.com/acme/my-harness.
Start with one Terminal-Bench task and one DeepSWE task.
The skill sets up a benchmark checkout for the task definitions and baseline results,
uses the Runta skills for tooling setup, then checks prerequisites and guides the
evaluation. Installing skills alone does not run evaluations. For a prompt with explicit harness, commit, provider, and build
settings, see PROMPT.md.
Already cloned this repository? Install the local copy from its root:
여기까지가 README 앞부분입니다. 나머지는 원문에서 읽을 수 있습니다. README 전문 보기 ↗
