verifiable-science-envs

Deterministic, executable-oracle RL environments and evaluation suites for clinical genomics — starting with HLA/immunogenetics.

Every answer is computed from the pinned IPD-IMGT/HLA release's own files. No human labels, no frequency data, no licensed tables — so the grader is auditable line-by-line, the sealed split regenerates on every release, and a model cannot have memorized the post-cutoff tasks.

The benchmarks

	Tasks	What it tests	Results
HLA-Bench-A	550	Nomenclature: truncation, expression suffixes, G/P groups, serology, rename history, null-allele and near-miss traps	bench/HLA-Bench-A.md
HLA-Bench-C	205	Donor–recipient matching: 6/6–12/12 frameworks, antigen vs allele level, hidden nulls, GvH/HvG direction, unresolvable typing	bench/HLA-Bench-C.md

Working on this repo? Read CLAUDE.md first: a push to main deploys production, and the project's status, decisions and runbook live in the private portfolio hub rather than here.

Headline findings so far: every model family tested (Claude, Qwen, Mistral, Llama, Phi, Gemma) scores 0% on 2-field ambiguity expansion (0 of 30 tasks per model on the full 550-task suite), the core clinical trap; models fabricate allele names at 0.06–0.20 per task, and the anthropic/claude-sonnet-4-6 figure is 0.11 per task (54 of 550 tasks with a hallucinated name, clean re-run, all 550 tasks answered); on matching, the naive string baseline falls from 28% (family A) to 0%, and open models reach 0–14% because they count matched loci instead of chromosomes. Full tables with Wilson CIs on the bench pages; current state in the bench pages below.

Every headline figure above is recomputed from the committed run artifacts in bench/reproduce.ipynb, which prints the published number next to the recomputed one with a pass or fail for each claim. It needs no API key and no local checkout.

HLA-Verify — the graders as an API

The same engine as a verification service (no LLM, no storage): POST /v1/verify checks every allele-shaped token in free text against the pinned release (fabricated / deleted-with-successor / valid, with G groups and flags — a legacy spelling such as A*0201, Cw*0702 or Cw*07:02 is valid by the current name it stands for, flagged deprecated_name, and a fabricated Cw*9999 is caught; NMDP MAC codes are surfaced as mac_code and KIR allele names as out_of_scope, never silently skipped or counted as fabricated); POST /v1/normalize fixes typing reports; GET /v1/allele/<name> returns the facts; POST /v1/match scores a donor–recipient pair under the published rules R1–R6.

Accepted as reported typing everywhere (/v1/verify, /v1/normalize, /v1/allele, /v1/match and the lab toolkit): current names at 1–4 fields, legacy spellings (colon-less A*0101, Cw*0702; the old Cw locus label on a colon-style name, Cw*07:02), deleted names (chased to their successor), G and P group names (A*02:01:01G, DPB1*04:01P — the resolution NGS reports are issued at), the XX code (A*02:XX), two-field A*02:01g, and an optional HLA- prefix. Each shorthand leaves a flag (deprecated_name, xx_code, lg_notation) so nothing is converted silently; GET /v1/allele/A*0101 answers with the facts of A*01:01 and resolves_to. Every gene in the release is checked (all 46 IPD-IMGT/HLA genes, HLA-E/F/G, MICA/MICB, DRA, TAP among them); matching frameworks cover A, B, C, DRB1, DQB1, DPB1. Not covered: NMDP multiple allele codes (A*02:AB, recognised but not expanded), serology-only typings (A2, DR4), KIR genes (IPD-KIR; KIR3DL1*001 in free text is recognised and reported out_of_scope, not checked and not counted as fabricated), and releases other than the pinned one. The pin is a git tag of IPD-IMGT/HLA (v3.66.0-alpha, dated 2026-10-14), which is ahead of the latest official release (3.65.0); /docs and GET /v1/releases (pin) say so, and the pin moves to the official 3.66.0 when it is published. The release field stays the bare 3.66.0.

Hosted, live: api.hlaverify.com (also https://hlaverify.com/v1/…). Open for evaluation at 100 calls a day per IP (60 requests/minute, up to 250 typings per /v1/normalize call); keyed access for labs, LIMS vendors and agent platforms with higher daily quotas and larger batches ([email protected]). Quotas reset at UTC midnight and every billable response carries x-hla-verify-daily-limit, -daily-remaining and -daily-reset.

curl -s https://api.hlaverify.com/v1/verify -H 'content-type: application/json' \
  -d '{"text": "A*0101, B*15:504:01, DQB1*05:03:26:99"}'

The hosted API is a Cloudflare Worker (edge/) that looks names up in tables exported from the pinned release by this repository's Python engine (python -m sci_envs.service.edge_export); a golden test (edge/test/) proves the Worker's output is byte-identical to the Python service on thousands of generated inputs.

Pharmacogenomics (hosted only): POST /v1/pgx/screen normalizes a typing first and then joins each two-field allele to the published CPIC and PharmGKB HLA–drug associations (B*57:01 → abacavir, B*15:02 and A*31:01 → carbamazepine, B*58:01 → allopurinol, …), POST /v1/pgx/pair does it for a recipient and a donor and says whose genome carries each actionable allele, and GET /v1/pgx/table (JSON or ?format=csv) is the whole table — free, no key. The table is hla-pgx-open (CC BY-SA 4.0), vendored at deploy time by python -m sci_envs.service.pgx_export into a bundle beside the release tables, never inside them; every response carries the table's snapshot date and commit next to the IPD-IMGT/HLA release. Guideline lookup only — decision support, not a medical device.

Self-host

From a checkout:

pip install -e ".[service]" && uvicorn sci_envs.service.app:app

Or build the container image yourself (same reference data fetch, same entrypoint):

docker build -t hla-verify -f sci_envs/service/Dockerfile .
docker run -p 8000:8000 hla-verify

Published images (from a tagged release) are at ghcr.io/jasonbrelsford/hla-verify once one exists:

docker run -p 8000:8000 ghcr.io/jasonbrelsford/hla-verify:latest

Live demo (runs entirely in your browser — typing data never leaves your machine): hlaverify.com/demo · mirrored on Hugging Face: Spaces/jason-brelsford/hla-verify

For AI agents: MCP server

Any MCP-capable agent can add HLA-Verify as a tool server and verify HLA content before presenting it (verify_text, normalize_allele, allele_info, match_score, check_typing, donor_compat, validate_gl_string, about) — as a remote server, or self-hosted over stdio (every tool except allele_info). The remote server speaks MCP 2026-07-28 (server/discover) and the legacy initialize handshake.

Remote (Streamable HTTP, JSON-RPC 2.0, stateless — nothing to install):

{"mcpServers": {"hla-verify": {"url": "https://api.hlaverify.com/mcp"}}}

Add "headers": {"Authorization": "Bearer YOUR_KEY"} for a keyed tier; anonymous calls share the free tier's 100 calls a day and 60 req/min. Works in Claude Desktop, claude.ai connectors, Cursor, and any other MCP-capable client.

Local (stdio):

pip install -e ".[mcp]"
python -m sci_envs.mcp_server        # stdio MCP server

Client config: {"command": "python", "args": ["-m", "sci_envs.mcp_server"]}. Also see skills/hla-verify/ (importable Claude skill) and hlaverify.com/llms.txt.

Run the benchmark

pip install -e ".[dev]"
pytest -q                             # first run fetches ~33 MB of reference data
hla-bench generate                    # family A (or --family b, --family c); sealed split stays local
hla-bench run baseline-naive-string --suite runs/hla-bench-a --split dev
hla-bench run ollama/qwen2.5:7b --suite runs/hla-bench-a --split dev
hla-bench run anthropic/claude-sonnet-4-6 --suite runs/hla-bench-a --split all
hla-bench report --suite runs/hla-bench-a --out bench/HLA-Bench-A.md
hla-bench run oracle --suite runs/hla-bench-b --split dev     # family B: 240 phasing tasks, grader dispatched on the task's family

Local models run free via Ollama; Anthropic/OpenAI/Gemini clients are included (keys via a gitignored .env). Raw responses and per-task scores never leave the machine; only aggregates and a stratified ≤3-per-subtype wrong-answer sample are committed.

Layout

sci_envs/
  reference/imgt.py         # pinned IPD-IMGT/HLA loader: fetch → md5-verify → query
  families/nomenclature/    # family A: generators, grader, normalizer
  families/phasing/         # family B: synthetic Mendelian truth generator + 8 frequency-free subtypes and grader (HLA-Bench-B)
  families/matching/        # family C: rules engine (R1–R6, documented for lab audit)
  harness/                  # runners, model clients, report
  adapters/                 # verifiers (Prime Intellect) + Inspect AI exports
  service/                  # HLA-Verify API: FastAPI service + edge table exporter (PolyForm-NC)
edge/                       # HLA-Verify API on Cloudflare Workers + golden test vs the Python oracle (PolyForm-NC)
environments/hla_nomenclature/   # pip-installable verifiers environment
harbor/                     # Terminal-Bench-style task
docs/                       # task + grader specs (families A, B, C)

Data strategy & partners

Every graded answer is computed from public, versioned data — the pinned IPD-IMGT/HLA release, synthetic Mendelian truth, and open population resources — so anyone can regenerate the suites and audit every score. Restricted registry data stays with its licensed holders: our environments run on their machines. We are seeking registry, lab, and model-developer partners — [email protected].

Licence

Open core: benchmark, generators, graders, harness, and adapters are Apache-2.0 (LICENSE). The HLA-Verify service (sci_envs/service/, edge/) is PolyForm Noncommercial 1.0.0 — free for research and evaluation; commercial use requires a licence from Brelsford Software LLC ([email protected]). Reference data are fetched at runtime from IPD-IMGT/HLA under CC-BY-ND (Barker DJ et al., NAR 2025) and never redistributed.

Scope: human clinical-genomics informatics only. No sequences, no pathogens, no wet-lab protocols.