Fynn Lab

A study of how to evaluate a skill-based LLM assistant: an eval set graded by an LLM judge, a regression check for every skill edit, a check of the judge against a person, and a plan for experiments on real traffic. The bots it evaluates are an independent re-implementation of the architecture change described in Kikoff's engineering post "Fynn at 100% scale".

It is a study project and is not affiliated with Kikoff. The products the bots talk about are fictional; see docs/products.md.

Site: youngsukhwang.github.io/fynn-lab. It has the eval framework and results pages. Its Demo page chats with both bots through a hosted backend with a small daily allowance (see "Hosted backend" below). Recorded replies from the committed eval run are one click away, and the page switches to them when that backend is asleep or not set up.

In short

• The routing change helps where a question needs a second domain, and the result held out of sample. On those questions the skill design passes 79% of runs and the single-agent design 16%; on a pre-registered held-out set written after the skills were frozen, 39 of 39 runs against 12 of 39 (9 of 13 cases better, none worse; sign test p = 0.004). When one domain is enough, the two are about even.

• Every skill edit gets a regression check. Only the cases it can affect rerun; each flip counts as fixed, broken or chance, depending on whether the edit could reach that case; and a decision stays a lead, not proof, unless an exact test on the cases it could reach backs it. None of the four edits shown is backed: with one run per case a Ship takes six fixed cases and none broken, and the best edit fixed five.

• The LLM judge was checked against a person. On 13 sampled replies graded blind (48 facts), it agreed with the person on 92% of facts, and in all 4 disagreements it was the stricter one.

• Cost and models. Sonnet came close to Opus at about 41% of the cost, and the skill design costs about 9% less per reply than the single-agent one.

• Before: the earlier design in the post, which the post calls its beta. A small model routes each message to exactly one domain expert agent, so a question that spans several domains gets only one domain's expertise.

• After: the skill design the post moved to. A small model selects the topic skills the message needs. One agent answers with those skills plus always-on skills (tone, response rules, guardrails).

Both bots read the same skill files, see the same member profile and use the same answer model. They differ only in routing and prompt composition.

Fynn Lab additions, not from the post: the After bot caps new skills at 4 per message (3 until the cap experiment below) and carries up to 2 of the previous turn's skills for one more turn; the products, personas and the four Before agents are this project's own.

What it found

Before vs After and the answer models ran three times per case, on the skills shipped 2026-10-23.

• After wins where a question needs a second domain, and only there. On the 33 cases whose skills span two or more Before domains, After passes 78 of 99 runs and Before 16. A Before agent holds only one domain's skills: in none of those runs did its prompt hold every labeled skill, and After's did in 92 of 99. On the 31 cases one domain covers, the two are about even (85 and 87 of 93).
• Most of that gap comes from cases built to show it. 24 of the 33 multi-domain cases (mi-013 to mi-036) were written for this comparison: every question beyond the Before router's likely pick was built around a fact only another domain's skill held. There Before passes 2 of 72 runs and After 55. Both Before passes are mi-016, whose fact a later skill edit (S4) copied into a skill the Before agent loads. On the other 9 multi-domain cases, which were not built around the Before domains, the gap is smaller and less certain (14 and 23 of 27 runs): only 4 of the 9 came out differently, all in After's favor (sign test p = 0.125), and two of those 4 were cases the shipped edits targeted.
• It held on questions the skills were never tuned to. 24 held-out cases were written after the skills were frozen, by agents that never saw an eval result, and committed with written expectations before either bot ran (the plan). On the 13 that need a second domain, After passed 39 of 39 runs and Before 12 of 39; on the 11 one-domain cases they were about even (After 30, Before 31 of 33). Overall After passed 69 of 72 runs and Before 43, a mean gain of 36 points per case. The run-permutation test (p < 0.001) shows only that the gap beats run-to-run noise on these questions; counting cases as a sample of questions, 9 came out better and 1 worse (sign test p = 0.021), and on the 13 needing a second domain 9 better and none worse (p = 0.004). Two of the three expectations failed: the multi-domain gap (+69 points) came out close to the designed cases' (+73) rather than the other multi-domain cases' (+33), and After did better on the held-out cases than on the main set, not worse. The check is small, and its cases were also written by LLM agents from the same skill files.
• On real traffic the gain depends on the mix. Nobody has measured how often members ask a question that needs a second domain. If 10% do, the expected gain is about 5 points using the 9 multi-domain cases not built for the comparison (a rough figure, since only 4 of them differ), 8 using all 33, or 4 using the held-out cases; at 30% it is about 12, 20 or 19. The total on this eval set (After 165 of 192 runs, Before 101; a mean gain of 33 points per case, p < 0.001 on a run-permutation test) reflects this set's mix, which leans on multi-domain cases by design, so it is not a forecast.

Edit	What changed	Decision
S1	score-changes body: give per-card utilization on a score drop	Lead: tradeoff
S2	response-rules: answers under 120 words instead of 250	Lead: do not ship
S4	credit-builder-account body: closed accounts can stay up to 10 years	Lead: ship (shipped)
Cap 4	the After router may pick 4 new skills instead of 3	Lead: ship (shipped)
S3, S5, S6	three skill descriptions widened to route more questions (in the git history)	Lead: ship (shipped)

Skill-edit results are from one run per case. A decision is backed when an exact test on the cases the edit could reach (with one run per case, fixed against broken) is significant; otherwise it is a lead to read, not proof. A tradeoff is always a lead.

Limitations

• Single-turn only. Every case is one message, so carrying skills between turns is not evaluated.
• Tuning, and a small held-out check. The main cases, the expected facts and the skill fixes were all written looking at the same eval set, so its gap includes tuning: before the shipped edits, a single run on 2026-10-22 gave After 42 and Before 31 of 64 (p = 0.019); a single run after them gave 54 and 29. The 24-case held-out check shows the same pattern, but it is small, and its cases were also written by LLM agents from the same skill files.
• The judge is checked against one person, on a small sample. One LLM judge (Opus) grades every reply; the human check covers 13 random replies from the answer-model runs, all from the After bot, labeled by the author.
• Synthetic setting. Six fictional members, fictional products, and a Before domain split that is this project's own; a different split would change which cases need a second domain. No real members use the bots (the hosted demo is anonymous and rate-limited), so the Online evaluation page is a plan with no data.

How it works

• Eval set. 64 single-message cases over 6 synthetic members, in four datasets: single-topic questions, multi-part questions (36, of which 32 need a second domain, 24 of them written for this comparison), use of the member's own data, and basic safety. Cases live in backend/evals/cases/; the written-for-the-comparison ones are marked designed: true.

• Scoring. A case passes only if both checks pass: code checks (for example no promised score gains, using the member's own figures) and an LLM judge (Opus) that looks for every expected fact in the reply.

• Judge check. evals.judge_validation draws a blind sample of graded replies for a person to label with the judge's own rules, regrades stored replies with other judge models, and reports how often each judge agrees with the person and with the original judge.

• Before vs After. Both bots answer every case; the report pairs the cases and tests the net change.

• Skill edits. A scenario in scenarios/<id>/ holds an edited copy of one skill, laid over skills/ for the run. Only the cases the edit can affect rerun: a body edit reruns the cases labeled with that skill or routed to it; a description edit, an always-on edit or a cap change reruns all. The report then counts a flip for the edit only when the edit could reach that case's prompt:

  • a body edit, when the edited skill was loaded on both sides (when only one side loaded it, the router changed the prompt too);
  • a description edit, when the edited skill entered or left the case's routing (a reordering alone does not count);
  • a cap change, when the routing changed and the cap mattered: the run with the lower cap filled it, or the run with the higher cap went past it;
  • an always-on edit, always.

  When the two runs also differ in skill cap, a body or description edit also reaches the cases where the cap mattered. With the Before bot, the edited skill's domain stands in for the skill. Descriptions and the cap are also in the router's instructions, so these rules assume such an edit acts only through the skills it adds or removes. A decision (ship, do not ship) is backed when an exact test on the reached cases is significant in its direction, and is a lead otherwise. With one run per case that test is McNemar on fixed against broken; with more runs it is the run-permutation test on every reached case, since the flag rule's improved and regressed labels are not equally likely when nothing changed.

• Answer model and judge. The After bot is run with each answer model (Haiku, Sonnet, Opus 5.5) and the stored replies are regraded by other judge models; sums up cost, cases passed, reply length and judge agreement. Every report also says whether each case's answering agent had every labeled skill and how long the replies are ().

The detailed design notes are in SPEC.md, which is written in Korean.

Running the evals

The runner calls the Anthropic API for the routers, the answer model and the judge, so it needs ANTHROPIC_API_KEY in backend/.env. The tests never call the API. Run the commands from backend/; results go to results/ at the repo root. The runner overwrites --out, so move a results file a committed report still compares to results/archive/ first.

cd backend

# Both bots on every case, three runs each (how results/before.json and after.json were made)
.venv/bin/python -m evals.run --bot before --repeats 3 --out ../results/before.json
.venv/bin/python -m evals.run --bot after --repeats 3 --out ../results/after.json

# The held-out cases (--cases runs another case set and records it in the results file)
.venv/bin/python -m evals.run --bot before --cases evals/holdout --repeats 3 --out ../results/holdout/before.json
.venv/bin/python -m evals.run --bot after --cases evals/holdout --repeats 3 --out ../results/holdout/after.json

# A skill edit on the After bot; --baseline picks the cases it can affect. A shipped scenario no longer runs.
.venv/bin/python -m evals.run --bot after --scenario s1 --baseline ../results/after.json --out ../results/scenarios/s1-new.json

# The After bot with another skill cap (the default is 4 since the cap experiment); every case reruns
.venv/bin/python -m evals.run --bot after --max-skills 5 --baseline ../results/after.json --out ../results/cap5.json

# Reports: report.json (for the site) and report.md. A report refuses to replace one of other runs,
# or to use a baseline that does not fit the scenario, without --force.
.venv/bin/python -m evals.report --baseline ../results/before.json --candidate ../results/after.json --out ../results/reports/before-vs-after
.venv/bin/python -m evals.report --baseline ../results/after.json --candidate ../results/scenarios/s1-new.json --out ../results/reports/s1-new

# After a fatal API error (a bad key, no credit) the run stops and writes <out>.partial.json; rerun only what errored
.venv/bin/python -m evals.run --bot after --repeats 3 --rerun-errors ../results/after.partial.json --out ../results/after.json

# The After bot with Haiku and with Opus answering, three runs each
FYNN_AGENT_MODEL=claude-haiku-5-5 .venv/bin/python -m evals.run --bot after --repeats 3 --out ../results/models/k3/after-haiku.json
FYNN_AGENT_MODEL=claude-opus-5-5 .venv/bin/python -m evals.run --bot after --repeats 3 --out ../results/models/k3/after-opus.json

# The answer model and judge report (no API). The regrades graded earlier single runs, passed with --regraded-runs.
.venv/bin/python -m evals.models_report --runs ../results/models/k3/after-haiku.json ../results/after.json \
  ../results/models/k3/after-opus.json --regraded-runs ../results/models/after-*.json \
  --regrades ../results/validation/regrades --out ../results/reports/models

# Judge check: draw a blind sample to label and its page (no API), regrade stored replies with another judge
# model (API), then compare (no API). A sample file is not overwritten without --force.
.venv/bin/python -m evals.judge_validation sample --runs ../results/models/after-*.json \
  --include after-sonnet:mi-023 --out ../results/validation/sample-new.json
.venv/bin/python -m evals.judge_validation page --sample ../results/validation/sample.json \
  --translations ../results/validation/translations.ko.json --out judge-check.html
.venv/bin/python -m evals.judge_validation regrade --results ../results/models/after-sonnet.json \
  --judge-model claude-sonnet-5-5 --out ../results/validation/regrades/sonnet-on-after-sonnet.json
# --labels is optional; without it, or before the file exists, the report has only the judge-to-judge comparison
.venv/bin/python -m evals.judge_validation agreement --sample ../results/validation/sample.json \
  --labels ../results/validation/human-labels.json --regrades ../results/validation/regrades \
  --runs ../results/models/after-*.json --out ../results/validation/agreement

Reproduce the committed reports

The reports need no API. Each skill edit compares the After run its scenario was run against, which has since moved to results/archive/. Rerunning a scenario itself needs the commit it was run at (ac8ccae for S1 and S2, f303797 for S4 and the cap), since S4 and the cap have shipped and the skills have changed.

cd backend
.venv/bin/python -m evals.report --baseline ../results/before.json --candidate ../results/after.json --out ../results/reports/before-vs-after
.venv/bin/python -m evals.report --baseline ../results/holdout/before.json --candidate ../results/holdout/after.json --out ../results/reports/holdout
.venv/bin/python -m evals.report --baseline ../results/archive/after-2026-10-22T17-15-15.json --candidate ../results/cap4.json --out ../results/reports/cap4
for s in s1 s2; do
  .venv/bin/python -m evals.report --baseline ../results/archive/after-2026-10-22T01-19-16.json \
    --candidate ../results/scenarios/$s.json --out ../results/reports/$s
done
.venv/bin/python -m evals.report --baseline ../results/archive/after-2026-10-22T17-15-15.json \
  --candidate ../results/scenarios/s4.json --out ../results/reports/s4

• Models report: the evals.models_report command above.
• Bundle comparison (12 fixed, 7 broken): --baseline ../results/models/after-sonnet.json --candidate ../results/archive/after-2026-10-23T19-06-17.json, the single runs before and after the shipped edits.
• Judge sample: the committed results/validation/sample.json (seed 0, sample_id 524a504be8710b63) comes out of the sample command at commit 53832be, with the arguments it records in args: the three results/models/after-*.json runs and one --include for each of the eight targeted replies. The facts of st-001 were reworded since, so at later commits that command refuses the st-001 includes.

Notes

• Cost. One run of the 64 cases cost about $1.70 to $1.85 per bot at list prices with Sonnet answering and Opus judging, as recorded in the committed results; the judge is about two thirds of it. Three runs per case cost three times as much (Before $5.54, After $5.18). Each case's result records the router's, the answering model's and the judge's tokens apart (routing.usage, usage, judge.usage); older results files add the router's tokens to the answer's. The billing console is the final word, and the runner does not ask before spending.
• Safety nets. Results record fingerprints of the skills, the bot and judge code, and the settings, so a run against a baseline reruns every case when any of those or the models changed. A report leaves out cases whose expected facts were reworded between the two runs.
• More statistics. backend/evals/stats.py handles repeated runs: --repeats K gives a stricter flag rule and the run-permutation test, which the site uses for Before vs After and the answer models. Its A/A noise floor and Holm correction appear only in the report Markdown. In simulations of the original 40-case set where nothing changed, the test calls about 3% of comparisons significant at 0.05; when eight cases truly drop, it catches the drop in roughly 75% to 95% of simulations with three runs per case, against 20% to 30% with one. See SPEC.md.

Layout

skills/      skill files (Markdown + YAML frontmatter), shared by both bots
personas/    synthetic member profiles (JSON, no real data)
scenarios/   skill-edit scenarios: an edited copy of one skill and what it should do
results/     committed eval results and reports (archive/ holds runs a report still compares)
docs/        product fact sheet for the fictional app
backend/     Python: fynn core package, FastAPI app, eval runner and reports, tests
web/         Next.js site: the demo and the eval pages
render.yaml  Render Blueprint for the hosted backend of the public demo

Backend

Requires Python 3.12 or newer.

cd backend
python3.12 -m venv .venv
.venv/bin/pip install -e ".[dev]"
cp .env.example .env        # then add your ANTHROPIC_API_KEY

.venv/bin/python -m pytest                                  # tests use a fake LLM, no key needed
.venv/bin/python -m fynn.chat --persona rebuilder           # chat with both bots in the terminal

# HTTP API for the demo site. The extra flags also reload when skills or personas change.
.venv/bin/uvicorn api.main:app --port 8000 --reload --reload-dir . --reload-dir ../skills \
  --reload-dir ../personas --reload-include '*.md' --reload-include '*.json'

Settings in backend/.env are loaded before anything else, and variables exported in your shell take precedence over them. The skill caps are in backend/fynn/config.py; evals.run --max-skills N overrides the new-skill cap for one run.

Role	Model
Routers (both bots)	claude-haiku-5-5
Answering agent	claude-sonnet-5-5, low effort
Eval judge	claude-opus-5-5

You can override each model with an environment variable; see backend/.env.example.

Endpoint	Purpose
POST /chat	Takes {bot, persona_id, messages, carry_skills?}. Returns the reply, the model, the routing decision and token usage. Errors come back as JSON {detail, retryable}.
GET /personas	Synthetic member profiles
GET /architecture	Domains, skills, models, skill budgets and request limits, for the site's diagrams

FYNN_CHAT_LIMIT_PER_VISITOR_PER_HOUR and FYNN_CHAT_LIMIT_PER_DAY cap /chat requests for a public demo (api/limits.py); they are unset locally, so there is no limit. A refused request gets a 429 with Retry-After. The counts live in memory and reset when the server restarts.

Hosted backend

The public demo's backend runs on Render's free tier, set up by the Blueprint in render.yaml: Render installs backend/ and starts the API, and redeploys on pushes to main that touch it. It allows 20 chat requests per visitor per hour and 400 a day in all (10 and 200 questions to both bots, about $4 to $5 of Sonnet answers), and only the GitHub Pages origin. The free instance sleeps after 15 idle minutes and takes a minute or two to wake; the site then shows recorded replies, keeps checking, and offers the live chat once the backend answers. The counts live in memory and start over each time it wakes, and a client can fake the address the per-visitor limit counts by, so they slow ordinary use but do not bound spending. The only hard cap is a spend limit in the Anthropic Console: put the demo's key in a workspace of its own and set that workspace's monthly limit to the most you are willing to spend.

One-time setup:

1. In Render, choose New, then Blueprint, sign in with GitHub and connect this repo.
2. When asked, enter ANTHROPIC_API_KEY: a key used only for this demo, created in an Anthropic Console workspace of its own. Set that workspace's monthly spend limit first; it is the only hard cap.
3. Once it is live, add the service's address (for example https://fynn-lab-api.onrender.com) as the GitHub Actions variable FYNN_API_URL, and run the "Deploy site" workflow so the site is rebuilt with it.

Web

Requires Node.js 20.9 or newer. The demo needs the backend on port 8000, and chatting needs ANTHROPIC_API_KEY in backend/.env.

cd web
npm install
npm run dev                 # http://localhost:3000

• Demo (/): pick a synthetic member and send one message to both bots; each column shows the reply, the router's decision and the skills used.
• Before vs After and Skill edits: each has a Framework tab (how it is measured) and a Results tab (what was found), built from the committed results and reports.
• Models: the answer-model comparison (cost against cases passed, reply length) and the judge check.
• Online evaluation: a framework page for evaluating the bots on real traffic.
• NEXT_PUBLIC_API_URL points the site at another backend (read at build time); FYNN_CORS_ORIGINS in backend/.env allows other origins. In npm run dev, reload a page to pick up new results files.
• Static site. FYNN_STATIC_SITE=1 FYNN_BASE_PATH=/fynn-lab npm run build writes plain files to web/out/; .github/workflows/pages.yml does this on every push to main and publishes to GitHub Pages, with NEXT_PUBLIC_API_URL from the repository variable FYNN_API_URL. Its Demo page also offers recorded replies, and switches to them when the backend does not answer within 15 seconds; with no API address it shows only those. They are results/demo/recorded.json, which .venv/bin/python -m evals.demo_snapshot --before ../results/before.json --after ../results/after.json --out ../results/demo/recorded.json (from backend/, no API) rebuilds from the committed runs.
• Checks. npm run typecheck and npm run lint.