wilco

wilco grades a model on your task's own exam and tells you the day it stops passing. Eight tasks, four grading modes, one trajectory mode: labels, fields and properties grade a single answer, trajectory grades the tool calls. Write the cases, let every candidate sit them, pick takes the cheapest that clears the bar, check re-sits the exam per case, grade scores answers made elsewhere.

step	command
1. write the exam	wilco new <task> --mode labels · wilco validate <task>
2. every candidate sits it	wilco diff <task> --models ollama/<model>,openrouter/<id> --allow-tier-breach
3. the cheapest that passes wins	wilco pick <task>
3b. adopt the winner	wilco adopt <task> <provider/model> (writes provider: + model: in agents/<task>/agent.yaml, then snapshots it: 3 loads at temperature 0); on a tier: local exam a hosted winner is refused unless --allow-tier-breach, and then gets no snapshot, only the same-day rerun
4. know the day it stops passing	wilco check <task> (exit 1 if a stable case moved)
grade answers made elsewhere	wilco grade <task> <answers-file>: answers produced elsewhere, graded by the same checks (e.g. in n8n; no model call; single-turn exams only; see integrations/n8n)
then: run it	wilco live <task> --input "..." (through routes.yaml)

wilco <command> --help lists each command's options; run, diff and check keep their research flags behind --help-advanced.

Results for the eight example tasks: roger3000.com/wilco. No developer, or no time to write the exam? Send us your task: roger3000.com/wilco/pricing.

Five minutes, from clone to a scored table

git clone https://github.com/Michiel-DK/wilco && cd wilco
pip install -e .                                                     # the `wilco` command; requests, PyYAML
wilco list                                                           # the eight example tasks, their champions
wilco run email-triage --provider stub --model stub

The exams live in the clone, not in the package, so install it editable from the clone. Without installing anything but pip install -r requirements.txt, every wilco <command> is python3 wilco_cli.py <command>; the shim runs scripts/exam_new.py, scripts/exam_validate.py, scripts/pick.py, scripts/exam_adopt.py, scripts/exam_grade.py and sandbox/runner.py unchanged.

The stub needs no network, no key and no model. It answers reply_later to every email, so it fails most cases. What you get is the shape: one line per case, train and held-out scored apart, a result file under results/. Then the pick rule over the committed 27 Sep files, also offline:

wilco pick crm-followup docs/probes/matrix-2026-09-27/results/crm-followup__*.json --allow-mixed-graders

It prints pick: openrouter/deepseek/deepseek-v4-pro-0813. Those files predate the verdict stamp (no grader_hash, so pick cannot tell which check version graded them and refuses to rank them without --allow-mixed-graders; it prints a warning naming each). That folder holds the eleven hosted runs only; the local champion's snapshot covers a larger case set (47 cases against the matrix's 38 today), and pick refuses to compare unequal case sets rather than intersect them. (pick over the stub's own result prints NONE: the stub has no price, and an unknown cost is never counted as 0.)

A hosted model. Put your key in the environment as OPENROUTER_API_KEY (nothing is read from a file) and run the same command:

wilco run email-triage --provider openrouter --model <id> --allow-tier-breach

The example exams are tier: local, so a hosted provider needs --allow-tier-breach (see the tier rule below).

A local model. Install Ollama, then build the crm-followup champion's 16k-window variant and run its exam:

./models/models.sh gemma4-e2b-ctx16k   # pulls the base unless on disk with the measured weights; --dry-run: plan
wilco run crm-followup --provider ollama --model gemma4-e2b-ctx16k

models.sh with no name builds all three in models/. It checks each base tag's weights against the ones the scores were taken on and refuses a tag that has moved: on 23 Oct gemma4:e4b had, so the non-champion gemma4-e4b-ctx16k-plain is refused unless you pass --allow-drift (and then its scores are a new measurement).

How it works, in four steps

1. Name the tasks you already send to a model. Triage, extraction, drafting, summaries. The only requirement is that a person can say, case by case, whether the answer was right. If nobody can check it, this suite will not score it and says so. How to write one: docs/write-an-exam.md.

2. Write down what a correct answer looks like. Twenty to fifty cases per task, each with its checks: the label, the fields, the facts that must appear, the tools that must be called. Part of them is held out (here 191 of 310) and never tuned on. Scoring is deterministic. No model grades another model's answer, anywhere.

3. The cheapest model that passes wins. Every candidate sits the exam. Models below the bar drop out. The bar is the best score any model reached, or one case from it. The cheapest model left takes the job, and its score is committed as the snapshot that every later change is checked against.

This is the beginning of the README. Read the rest at the original. Read the full README ↗