TraceLens / 迹镜

TraceLens is a friendly evaluation and regression-testing framework for AI agents. It turns agent runs into inspectable traces, graded outcomes, baseline comparisons, and CI-ready reliability signals.

迹镜是一个面向 AI Agent 的评测与回归检测框架。它把每次 agent run 转化成可观察的轨迹、可评分的结果、可比较的 baseline，以及可用于 CI 的可靠性信号。

📖 Documentation: https://ssf0409.github.io/tracelens/  •  📦 PyPI: pip install tracelens

Why TraceLens

Agents are non-deterministic, so "the tests pass" says little about whether an agent change is safe to ship. TraceLens gives a Python team a regression check that lives in its own repository:

• Repo-owned, local, no backend. Tasks, adapters, graders, baselines, and a tracelens.yaml are files you commit; runs write JSON, Markdown, and HTML artifacts next to them. Nothing needs an account or a server, and CI is a plain job running the same command you run locally.
• Inspectable evidence. Every run keeps its trials, transcripts, grader feedback, and provenance (which task content, graders, and settings produced the numbers, and which candidate was under test). tracelens inspect explains a failure from those files.
• Explicit uncertainty. pass@k and pass^k separate capability from reliability, intervals come from a task-level bootstrap, numbers that were not measured are reported as unavailable rather than zero, and a gate that cannot be evaluated says so instead of passing.
• Harness failures stay separate from agent failures. Infra errors and grader crashes are counted on their own, so a broken eval never looks like a regression.

Use it when you need to answer questions like:

• Did this agent produce the right outcome, not just run without crashing?
• Is a flaky success still a real capability after 3–5 attempts?
• Did a prompt, model, tool, or infra change regress a baseline?
• Can CI block unsafe or lower-quality agent behavior before it ships?

It supports both subjective evaluation (LLM-as-judge for quality) and objective evaluation (schema validity, tool-use constraints, latency, budget, or domain-specific metrics).

What is demonstrated today

Each claim above rests on a different kind of evidence; here is which:

Claim	What it rests on
The documented workflow works from a fresh install	A CI job installs a freshly built wheel into a clean environment and drives init, run --config, baselines, the gate, an intentional regression, inspect, compare, a targeted rerun, an infra outage, a grader crash, malformed input, and checkpoint/resume through the console script.
The statistics do what the contract says	Hand-derived and independent-reference tests against the statistical contract (task-level bootstrap with multiplicity, order-independent pass^k, paired run comparison).
Integrations behave at their boundaries	Tests exercise the JSON/JSONL/CSV loaders, the HTTP adapter, the optional Hugging Face loader, the generated GitHub workflow, and tracelens.yaml.
TraceLens caught a real regression in a real project	Not yet published. The examples and the scaffold use simulated agents. A sanitized downstream case study is the open half of issue #33; until it exists, treat "catches regressions" as a tested mechanism, not an observed result.

Hosted evaluation and observability platforms also run datasets, experiments, and CI checks; the difference is where the evidence lives and what is required, not whether evaluation exists. See TraceLens vs Adjacent Tools.

Install

# Recommended: uv
uv pip install tracelens

# Or: plain pip
pip install tracelens

For the repository examples and local development tools:

git clone https://github.com/ssf0409/tracelens.git
cd tracelens
uv pip install -e ".[dev]"

See Installation for extras ([llm], [http], [datasets]) and CI setup.

5-Minute Demo

python examples/hello_world.py
tracelens report --results examples/reports/hello_world_report.json --format markdown

Expected first output:

tracelens hello-world
--------------------
trials run : 9
pass rate  : 100%
report json: examples/reports/hello_world_report.json
sample md  : examples/reports/hello_world_report.md

The checked-in sample report shows the concrete pieces a real eval needs: tasks, trials, pass@k, pass^k, graders, baseline comparison, regression result, and CI summary.

To start inside your own project:

tracelens init .
tracelens run --config tracelens.yaml

tracelens init writes user-owned starter files under eval/, a tracelens.yaml holding the run settings, and a GitHub Actions workflow that runs the same command on every pull request. Flags on the command line override the file. It refuses to overwrite generated files unless you pass --force.

What an eval looks like

Four pieces — Task, Adapter, Grader, Runner — and a report:

import asyncio
from tracelens import (
    Task, EvalSet, SimpleAdapter, CodeGrader,
    EvaluationRunner, RunnerConfig, Transcript,
)
from tracelens.reporting.generator import ReportGenerator

# 1. Define tasks
eval_set = EvalSet(name="Math Suite", tasks=[
    Task(name="Add 2+3", input_data={"a": 2, "b": 3}, metadata={"expected": 5}),
    Task(name="Add 10+20", input_data={"a": 10, "b": 20}, metadata={"expected": 30}),
])

# 2. Wrap your agent
async def math_agent(input_data: dict) -> dict:
    return {"answer": input_data["a"] + input_data["b"]}

adapter = SimpleAdapter(math_agent)

# 3. Write a grader
class MathGrader(CodeGrader):
    def compute_metrics(self, transcript: Transcript, task: Task) -> dict[str, float]:
        return {"correct": float(transcript.final_output["answer"] == task.metadata["expected"])}

    def determine_pass(self, metrics: dict[str, float], task: Task) -> tuple[bool, float]:
        return metrics["correct"] == 1.0, metrics["correct"]

# 4. Run and report
batch = asyncio.run(EvaluationRunner(adapter, [MathGrader("math")], RunnerConfig(num_runs=3)).run(eval_set))
print(ReportGenerator().render_markdown(ReportGenerator().build_report(batch)))

Walkthrough: Getting Started (5 min). Ready for a non-toy agent? Evaluating a Real Agent.

Documentation

The full, searchable docs live at https://ssf0409.github.io/tracelens/. Highlights:

Start here	Concepts	Guides
Is TraceLens For You?	Core Concepts & Glossary	Evaluating a Real Agent
Getting Started (5 min)	pass@k vs pass^k	Baseline Regression Tutorial
TraceLens vs Adjacent Tools	Accuracy Best Practices	Human-Eval Calibration
Installation	Multi-Level Evaluation	CI/CD Integration

Also: Build Your First Eval · User Guide · Loading Task Data · Evaluation Recipes · API Reference · Examples · Roadmap · Contributor Testing · Releasing.

Contributing

TraceLens is MIT licensed and open to contributions. Start with CONTRIBUTING.md, then run the local verification gate:

make verify   # lock check -> lint -> typecheck -> tests + coverage

Security issues should be reported privately using SECURITY.md.

Key Design Principles

1. Grade outcomes, not execution paths — focus on what the agent produced.
2. Handle non-determinism — pass@k for capability, pass^k for reliability.
3. Start with 20–50 real failure cases — build suites from actual issues.
4. Read transcripts regularly — catch false signals and grader bugs.
5. Calibrate with human evaluation — LLM graders drift without it.
6. Separate harness failures from agent failures — track infra/grader error rates alongside pass rates.

Informed by Anthropic's Demystifying Evals for AI Agents.