Hydrant on the Openbenchmarks company-enrichment benchmark

This repository has what it takes to run Hydrant through Openbenchmarks' company-enrichment benchmark with their own runner and judges, and our answers for all 282 companies of their cohort.

It states no score. The benchmark's number is theirs to give, and their judges are LLMs whose verdicts we can't stand in for. Anyone with the files here can score the answers themselves.

What's here

File	What
hydrant.py	The adapter, in the shape of their other providers. It goes into their scripts/firmographic/providers/
test_hydrant_adapter.py	Its tests, in the style of their adapter tests
answers/	Our 282 answers, one cell of their runner per company: the scored fields, the domain, latency and usage
reproduce.py	Clones their repository at the pinned commit, adds the adapter, and loads answers/ into their snapshot
convert.py	How answers/ was made from our saved API responses, through the adapter's own code

Nothing of Openbenchmarks' is copied here. Their repository has no licence, so reproduce.py fetches it from GitHub at commit bd4624c, their latest (2026-09-04), which holds the frozen 282-company cohort the board scores.

How the adapter calls Hydrant

The adapter sends POST /v1/enrichments with the domain and wait_seconds: 30. The API answers 200 if the enrichment ends within that, and 202 with a status_url if it's still running, which the adapter then reads every 2 s until it ends.

• Afresh every time. The adapter enriches each company again, whatever Hydrant already holds. It doesn't use GET /v1/companies/enrich, which answers from a stored record fetched in the last 30 days when there is one. We had already enriched these 282 companies, so that endpoint would have answered them in under a second at the price of a read.
• Latency is the whole wait, from the POST to the end, polling included.
• Usage is one hydrant_enrichment per company found. An enrichment that finds none of the five scored fields is given back, so it counts as 0. An error cell records no usage.
• Price: $0.0196 an enrichment, on the entry paid self-serve plan (Starter: $49 for 25,000 credits, at 10 credits an enrichment). See pricing.
• Errors: our own failure (enrichment_failed) is raised as a 503, which is how our API answers it elsewhere, so it becomes a retriable error cell. The adapter gives up after 15 minutes with a timeout, which is retriable too.
• Names aren't returned. The company's names aren't scored, and a sole trader's name is a person's, so the adapter leaves legal_name and aliases empty.
• Only the values. The audit holds the enrichment's id, mode and timestamps, not the sources behind each value.

Docs: docs.hydrantapi.com, and enrichments for the POST and its 202.

Load our answers into their snapshot, without calling us

python3 reproduce.py setup
python3 -m venv .venv && .venv/bin/pip install -r upstream/requirements.txt
(cd upstream && PYTHONPATH=scripts ../.venv/bin/python -m unittest firmographic.test_hydrant_adapter)
.venv/bin/python reproduce.py load

load copies answers/ into their runner's checkpoints and runs their runner for hydrant. It finds every company already answered, so it makes no call. The rows land in upstream/data/latest-firmographic.json, scored by their coded rules. Their runner ends with needs_retry at this commit, with or without us: its registry names Predict Leads predictleads and its snapshot names it predictleads-enrichment, so 282 of their own cells look missing.

setup also changes one of their tests. test_full_runner counts 8 providers (80 cells for 10 cases), and with Hydrant registered there are 9, so it now counts the registry instead.

Judging

Their board's numbers come from their LLM judges, not from the coded rules above. Their scripts/run_dedicated_field_judges.py --confirm-paid --apply judges each field with gpt-5.6-terra and needs an OPENAI_API_KEY. It judges every provider whose verdicts aren't saved in their repository, which at this commit is all of them except Seltz and ZoomInfo. Employee bands on the board are judged again with gpt-5.6-sol, by a script that reads their own database, so a run outside Openbenchmarks can't repeat that step exactly.

Run it live

You'll need a live key (hyd_live_…, from the dashboard at app.hydrantapi.com) that can enrich, on a paid plan. The free plan's daily credits run out after about 25 enrichments, and then a 402 stops the provider. Put it in upstream/.env.local as HYDRANT_API_KEY. Start from a fresh setup, deleting upstream/ first if load has run there: the runner skips companies it already has answers for.

cd upstream
PYTHONPATH=scripts ../.venv/bin/python scripts/run_firmographic_full_benchmark.py --only hydrant --confirm-paid

Their runner calls one company at a time. 282 enrichments take about two hours and cost about $5.53 at the entry plan's rate, less the misses given back.

How the answers were made

• Run: prod, 2026-10-23, release v0.15.1, through the public API with a live key, as the adapter calls it: POST /v1/enrichments, then polling. Later releases haven't changed the pipeline.
• Concurrency: 8 companies at a time, the concurrency of their Parallel run, so the latencies are higher than one at a time. The median here is 28.5 s. Twenty lone calls the same day, timed the same way, had a median of 22.7 s.
• Outcome: all 282 succeeded. 281 returned at least one scored field.
• Processing: each saved response went through hydrant.result_from, the code a live run uses (convert.py). The values are as the API returned them. Only the scored fields and the domain are kept, not the full record.

What we knew of the cohort beforehand. 32 of the 282 companies are also in our own golden set. We labelled them ourselves, tuned the pipeline on them, and trained our headcount estimator on them. We have run the cohort itself only to measure: no rule, prompt, threshold or training row comes from its references.

Licence

The files in this repository are under the MIT licence. Openbenchmarks' code and data aren't included, and they stay under their terms.