roaster-rank

Crawls specialty coffee roasters' online shops and builds a catalog of their whole-bean coffees: name, origin, process, roast level, description, and price per kg for every bag size.

How it works

roasters.toml ──crawl──▶ products (raw, per roaster) ──normalize──▶ beans + sizes ──export──▶ JSON / CSV

1. Crawl (deterministic, no LLM). One adapter per storefront platform pulls the full catalog and stores each product untouched (products.raw_json):

   • shopify: the public /products.json endpoint, optionally limited to one collection with path (49 of the 55 roasters)
   • woocommerce: the public Store API, with variations fetched in a second listing
   • squarespace: ?format=json on the shop page
   • wix: the store sitemap, plus the product JSON Wix embeds in each page
   • dispatch: Dispatch's headless Shopify, read from the Next.js __NEXT_DATA__
   • cafeunion: Café Union's custom shop, scraped from its product pages

   Each crawl upserts every product. Products missing from a successful crawl get retired_at set, and they come back automatically if they reappear. A failed crawl changes nothing. A crawl that would retire more than half of a catalog is refused (--allow-mass-retire overrides), because that usually means the site changed and the adapter broke.

2. Normalize (LLM). Products that are new, or whose content changed, go through:

   • a cheap keyword prefilter that drops obvious non-coffee (merch, gear, gift cards, subscriptions, wholesale-only items, $0 placeholders), then
   • Claude Haiku through the local claude CLI, with a strict JSON schema. It decides whether the product is a single whole-bean bag (samplers, subscriptions, capsules and so on are excluded), and extracts the name, origins, process, roast level, varieties, tasting notes and an English description. When the roaster doesn't state a roast level, the model infers one from the tasting notes and description, and roast_level_source records whether it was stated or inferred.

   Price and stock changes don't trigger a new LLM call. Only title, description, tags and variant labels count (RawProduct::content_hash).

3. Sizes are computed in code on every crawl. Grind variants collapse into one size, the cheapest in-stock price wins, and price per kg is derived from the weight. The weight is parsed from the variant label ("300g", "2 lb", "2,27kg - 5lb"). When the label has no weight, the LLM reads it from the description. Failing that, the platform's variant weight is used if it's a standard bag size, and then the roaster's default_weight_g. Each size records its weight_source. Variants matching exclude_variants (e.g. coffee sold inside a canister) are ignored.

Usage

cargo run --release -- run              # crawl + normalize everything
cargo run --release -- crawl --only nordik,zabcafe
cargo run --release -- normalize --limit 20 --concurrency 8
cargo run --release -- normalize --force --beans-only   # re-run coffee bags after a prompt change
cargo run --release -- status
cargo run --release -- export --format csv --out exports/beans.csv
cargo run --release -- export --format json --include-retired
cargo run --release -- export --format site --out site/index.html

Website

export --format site bakes the beans into site/template.html and writes a single self-contained site/index.html. It needs no server or build step: open it directly, or deploy it to any static host. The page has:

• a search box, plus filters for roaster, origin, process, roast, type, maximum $/kg, in-stock only, and stated-roast-only
• sortable columns, including $/kg for the smallest bag and the best $/kg across sizes
• details on row click: description, producer, varieties, every size
• filters and sort kept in the URL, so a filtered view can be shared
• light and dark themes, and a card layout on phones

The template is compiled into the binary, so rebuild after editing it.

Normalization shells out to claude -p --model haiku, so you need to be logged into Claude Code. Use --model to pick another model and --claude-bin to point at a different binary.

Adding a roaster

cargo run -- detect https://new-roaster.com    # prints a [[roaster]] entry

Paste the entry into roasters.toml. Detection identifies the platform and checks whether /en/ serves translated content. Sites on an unsupported platform need a new adapter in src/adapters/.

Data

SQLite at roaster-rank.db:

• products: raw catalog rows with first_seen, last_seen and retired_at
• beans: one row per screened product. is_coffee_bag = 0 rows record why the product was excluded.
• sizes: current bag sizes, prices and price per kg
• crawls: log of every crawl attempt