Project Eval

Measure what an AI agent actually delivers: tasks done, time spent by the agent and by you, cost, how often its work is kept or fixed, and quality scored by a judge that is checked against people. Every number says how it was obtained (measured, inferred or estimated), and hours saved are only shown when you give a baseline, never guessed.

Site: https://krishantm.github.io/project-eval — try it in the browser with your own logs, and see how the pipeline works step by step.

Three ways to use it

	What you need	What you get
1. In the browser	Nothing. Open the site and drop in a file	A report on your Claude Code or Codex sessions, or a CSV of agent inputs and outputs. Parsed on your machine; nothing is uploaded
2. On your machine	Docker	The full server: live telemetry, history, grading with a local judge, review queue, saved reports
3. For a team	A Kubernetes cluster	The same server on Helm, with workers that scale on the grading backlog and a network policy that keeps data inside the cluster

2. On your machine

ollama pull qwen3:8b          # the default judge, local; any LangChain model id works via JUDGE_MODEL
docker compose up --build

Open http://localhost:8000. To bring in your past Claude Code sessions:

docker compose run --rm -v ~/.claude/projects:/logs:ro api python -m eval_server.cli import-claude-code --root /logs

To send live telemetry from Claude Code, copy the environment block from the setup page (/v1/setup/snippets) into ~/.claude/settings.json. Remove it to stop.

3. For a team (Helm)

# build the image and push it where the cluster can pull it (here, a k3d registry on localhost:5050)
docker build -t localhost:5050/project-eval:1.0.0 . && docker push localhost:5050/project-eval:1.0.0
helm install eval deploy/helm/project-eval -n eval --create-namespace \
  --set judge.ollamaBaseUrl=http://<judge-host>:11434 \
  --set 'networkPolicy.judgeEgress[0].cidr=<judge-ip>/32' --set 'networkPolicy.judgeEgress[0].port=11434'
scripts/egress-check.sh eval http://<judge-host>:11434/api/version

Set keda.enabled=true to scale workers on the job stream (needs KEDA in the cluster). deploy/helm/project-eval/values.yaml lists every setting.

How it works

agent ──OTLP / import / API──> api ──> Postgres (tasks, steps, grades, reports)
                                │
                                └─ outbox ──> Redis stream ──> worker ──> judge (local model)
                                                                  │
                                                                  └─ sampled or unsure ──> person ──> calibration

• Attach. Claude Code sends OpenTelemetry logs and metrics with no code change; past sessions are imported from its log files; anything else posts tasks to /v1/tasks. Signal definitions: docs/SIGNALS.md.
• Store. One task per prompt, upserted so retries and out-of-order events never duplicate. Identity attributes (email, account and organisation ids) are dropped on arrival. Prompt and output text are kept only when content capture is switched on for that agent.
• Grade. A LangGraph flow: the judge scores against the agent's rubric, then a person scores a sample blind. The sample starts at 30% and falls to 5% once judge and people agree on 20 or more tasks. Grades pause in Postgres while they wait for a person, so a restart loses nothing.
• Report. Volume, speed, your time, cost, quality and value, each with its basis and, for sampled numbers, a 95% interval.

API: docs/API-v1.md. Design and decisions: docs/DESIGN-v1.md.

Development

python3.12 -m venv .venv && .venv/bin/pip install -r requirements.txt
brew install postgresql@17 redis     # the tests start their own throwaway servers
.venv/bin/python -m pytest
cd web && npm ci && npm run dev      # the site and app UI

The tests use synthetic data only and mock every model call.

Repository

Path	What
eval_server/	API, OTLP receiver, importer, grading graph, worker, report
migrations/	Alembic migrations (a test fails if models and migrations drift)
web/	Next.js app: the public site and the self-hosted UI, one static build
deploy/helm/	Helm chart: api, worker, Postgres, Redis, migration job, network policy, KEDA
eval_kit/, learn/, examples/	v0: a SQLite prototype and the LangChain/LangGraph learning scripts it grew from

Status

v1: browser evaluator, Claude Code import and live telemetry, business-agent rows, grading with human calibration, reports, Docker Compose and Helm. Planned: an LLM proxy for agents without telemetry (LiteLLM), Codex live telemetry, and an SDK that emits OpenTelemetry.

Licence

Apache-2.0