TraceBench

TraceBench checks recorded model outputs against a suite of test cases, so a prompt or model change can be reviewed before it ships. It joins cases to outputs by ID, runs literal checks, totals the latency and token cost you recorded, and fails a CI job when the candidate regresses against a saved baseline. Paired statistics help tell a regression from run-to-run noise.

It runs offline on Python 3.11 or newer, uses only the standard library, and never calls a model provider.

Try it

From a checkout:

python3 -m venv .venv
.venv/bin/python -m pip install --no-index .
mkdir -p reports
.venv/bin/tracebench evaluate --cases fixtures/cases.jsonl --outputs fixtures/baseline.jsonl --pricing fixtures/pricing.json --report reports/baseline.json
.venv/bin/tracebench evaluate --cases fixtures/cases.jsonl --outputs fixtures/clean.jsonl --pricing fixtures/pricing.json --baseline reports/baseline.json --report reports/clean.json
.venv/bin/tracebench evaluate --cases fixtures/cases.jsonl --outputs fixtures/regression.jsonl --pricing fixtures/pricing.json --baseline reports/baseline.json --format html --report reports/regression.html

The commands exit 0, 0 and 3. The bundled suite has six English and Mandarin cases with synthetic outputs, latencies and prices. The clean candidate passes all six, with its outputs in reverse order to show the join by ID. The last candidate breaks the English greeting and the Mandarin location check, so TraceBench reports two regressions and exits 3. Open reports/regression.html in a browser, or the hosted sample:

The same report is committed as HTML and as Markdown.

Inputs and checks

Cases and outputs are JSONL files, one object per line:

{"id":"greeting","input":"请回答你好","checks":[{"type":"exact","expected":"你好"}]}

{"id":"greeting","output":"你好","metrics":{"latency_ms":120.5,"input_tokens":10,"output_tokens":2}}

Check	Passes when
exact	The output equals the expected string, including case, whitespace and Unicode code points
contains	The output contains a nonempty literal substring
json_subset	A structured output contains the expected object; arrays keep their length and order, and booleans differ from numbers

A case passes only when every check passes. Pricing is a separate file with both rates as decimal strings per million tokens, for example {"currency":"USD","input_per_million":"0.10","output_per_million":"0.30"}. Inputs are validated strictly: unknown fields, duplicate IDs and non-finite numbers are errors. docs/SCHEMAS.md has the full contracts.

Choose a gate

On a nondeterministic system some cases flip between runs even when nothing changed, so failing on any single regression mostly reports noise. With a baseline, every report includes paired statistics: an exact McNemar test on the cases that changed status, bootstrap intervals for the change in pass rate, latency and cost, and the smallest change the suite could detect. --gate chooses what exit 3 means:

--gate	Exit 3 when
any-regression (default)	Any case passed in the baseline and fails now
mcnemar	The exact two-sided McNemar p-value is below --alpha and regressions outnumber improvements
noninferiority	The pass-rate interval's lower bound is below -margin, so the candidate has not shown it is within the margin
confident-regression	The pass-rate interval's upper bound is below -margin, so the drop is confidently larger than the margin

The paired demo in fixtures/paired/ has 100 synthetic cases, and its baseline passes 80 of them:

.venv/bin/tracebench evaluate --cases fixtures/paired/cases.jsonl --outputs fixtures/paired/baseline.jsonl --pricing fixtures/pricing.json --report reports/paired-baseline.json
.venv/bin/tracebench evaluate --cases fixtures/paired/cases.jsonl --outputs fixtures/paired/noisy.jsonl --pricing fixtures/pricing.json --baseline reports/paired-baseline.json
.venv/bin/tracebench evaluate --cases fixtures/paired/cases.jsonl --outputs fixtures/paired/noisy.jsonl --pricing fixtures/pricing.json --baseline reports/paired-baseline.json --gate mcnemar
.venv/bin/tracebench evaluate --cases fixtures/paired/cases.jsonl --outputs fixtures/paired/regressed.jsonl --pricing fixtures/pricing.json --baseline reports/paired-baseline.json --gate mcnemar --format markdown

Candidate	Pass-rate change (95% interval)	McNemar p	Default gate	--gate mcnemar
Noisy: swaps 5 passing and 5 failing cases	0 (-0.06 to 0.06)	1	exit 3	exit 0
Regressed: breaks 12 cases and fixes 1	-0.11 (-0.18 to -0.04)	0.00341797	exit 3	exit 3

The commands exit 1, 3, 0 and 3; the baseline itself exits 1 because it has failing cases. The default gate fails the noisy candidate on its 5 regressions, and the McNemar gate does not. At 80% power, this suite can only detect pass-rate changes of about 0.09. That is why --gate noninferiority --margin 0.05 rejects even the noisy candidate.

Gates judge pass/fail only, and they assume cases are independent. Suppose 10% of the cases in an unchanged 100-case suite flip at random. The default gate then fails 99.4% of the time, and the McNemar gate 1.1%. For small suites, prefer mcnemar. docs/METHODOLOGY.md has the formulas, the references and a simulation of each gate's error rates.

Use it in CI

Exit	Meaning
0	All checks pass and there is no regression, or a statistical gate passed
1	Some cases fail, but none regressed (default gate only)
2	Invalid input, an incompatible baseline, a usage error or an I/O error
3	A pass-to-fail regression, or a statistical gate failed

A baseline is a prior JSON report for the same suite. TraceBench checks a fingerprint of the suite, so changed prompts or checks need a fresh baseline. The reusable workflow uploads JSON, Markdown and HTML reports and then applies the gate; this repository's CI calls it with clean and regressed fixtures. tracebench policy adds optional budgets for pass rate, p95 latency and total cost. docs/CLI.md is the full command reference.

Docs

• Command-line reference: options, outputs, exit codes, baselines and budgets.
• Methodology: how each check and statistic is computed, and their limits.
• Library API: a complete offline example.
• CI integration: the reusable workflow, its artifacts and what it trusts.
• HTML reports: the offline report, its security boundary and its browser tests.
• Architecture: data flow, validation, baseline compatibility, packaging.
• Verification: what CI runs on every push, and how to run it locally.
• 中文指南: 快速运行与结果解释。

Development

python3 -m unittest discover -v

The tests cover file boundaries, multilingual checks, hand-computed metrics, malformed baselines, report escaping, exit codes from real subprocesses, and wheel and source-archive builds. The statistics tests compare exact p-values with hand derivations and brute-force enumeration, check the bootstrap's invariants over seeded random suites, and measure each gate's error rates by exact computation and simulation. CI runs them on Python 3.11 and 3.13, along with the recorded-evaluation gates and a Chromium job for the HTML report. See CONTRIBUTING.md and SECURITY.md.

Limits

• Checks are literal: there is no judge of meaning, facts, safety or language quality.
• TraceBench uses the latency, token counts and prices you supply; it does not time a provider or look up prices, and a cost over partly priced cases is not a total bill.
• The statistical gates assume independent cases, use one recorded run per report, and do not correct for running a gate on every push.
• Reports do not keep case inputs or expected values.
• The bundled data is synthetic, and this is a portfolio project that has not been used in production.

Built with AI coding assistants. MIT licensed.