Product Finder

Crawls a brand's storefront, reconstructs its full product hierarchy — categories, model families, generations, thicknesses, colourways, SKUs — and records price, discount and stock history so trends accumulate over time.

Phase 1 target: JOOLA · United States · pickleball paddles. Built to add countries (India, Vietnam, Australia) and brands as config, not code.

Verified against joola.com: 234 collections · 85 paddle products · 129 SKUs.

---

Quick start

1. Try the scraper with no database

Works immediately — no Supabase needed. Fetches the live catalogue and prints what the normalizer resolved.

npm install
npm run crawl:dry

Expected output:

dry-run: product lines  {"Perseus":19,"Hyperion":18,"Scorpeus":14,"Agassi":13,...}
dry-run: generations    {"Pro IV":36,"(unresolved)":22,"Pro V":12,"3S":12,"Gen 1":2}
dry-run: skill tiers    {"pro":58,"recreational":10,"performance":9,"unknown":7}
dry-run: thicknesses    {"16mm":63,"14mm":36,"10mm":12,"(none)":8,"12mm":4}

(unresolved) generations are correct — Vision, Edge, Champion, Dash and Beacon are genuinely generation-less lines.

2. Create the database

Create a Supabase project, then apply the schema by one of these routes.

A — script (no CLI login needed). Put the direct connection URI in .env as SUPABASE_DB_URL (Settings → Database → Connection string → URI), then:

npm run migrate:dry   # list what would be applied
npm run migrate       # apply; records each version in schema_migrations

Each migration runs in its own transaction, so a failure rolls back cleanly and re-running is safe.

B — Supabase CLI. Requires being logged in as an account that owns the project:

supabase login
supabase link --project-ref <your-ref>
supabase db push

C — dashboard. Paste supabase/schema.sql (all migrations concatenated, in order) into the SQL editor and run once.

Then set the API credentials in .env:

cp .env.example .env
# SUPABASE_URL                = https://<ref>.supabase.co
# SUPABASE_SERVICE_ROLE_KEY   = the SECRET key (sb_secret_... or legacy service_role)

The publishable/anon key will not work for ingest. RLS is enabled deny-by-default in ..._rls_and_seed.sql, so only the service-role identity can read or write. Keep the secret key out of git — .env is already ignored.

Confirm the seed landed:

npm run sites
# JOOLA      US  https://joola.com  shopify  15 collections

3. First real crawl

npm run crawl
npm run report

The first run is the baseline — it writes snapshots but no change events, because there is nothing to compare against. From the second run onward, variant_events fills with price moves, discounts and stock changes.

4. Schedule it

The history only exists if this runs regularly. A missed day is a permanent hole.

Local (Windows Task Scheduler) — registration block is in the script header:

powershell -ExecutionPolicy Bypass -File scripts\crawl.ps1

CI (GitHub Actions) — .github/workflows/crawl.yml runs daily at 09:00 UTC. Add SUPABASE_URL and SUPABASE_SERVICE_ROLE_KEY as repository secrets. Preferred once the repo is pushed: it does not depend on your PC being on.

---

The dashboard

npm run dev      # http://localhost:3000
npm run build && npm run start

Page	Shows
/	KPIs, deepest discounts, recent movement, Pro tier by generation
/catalogue	Every model, filterable by tier / generation / line
/models/[id]	Versions, thicknesses, colourways, athletes, SKU table, price history
/reviews	Every review, searchable — full text, star filter, complaints-only, verified-only, brand-replied-only, with the reply chain inline
/discounts	Everything on promotion, deepest first
/changes	The derived change log
/pipeline	Run history, errors, and a Run crawl now button

Pages are server components reading Supabase with the service-role key. The browser never talks to Supabase directly, so RLS stays deny-by-default and the secret key never reaches the client.

The price chart is hand-rolled SVG on purpose. Charting libraries interpolate linearly, which would draw a smooth slide between two observations and imply prices that never existed. observed_at is when we looked, so the only honest render is a step.

Before management sees it

• The site has no authentication. Anyone with the URL sees your pricing intelligence. Put it behind Vercel password protection, Cloudflare Access, or an auth layer before sharing the link outside the team.
• Set CRAWL_TRIGGER_TOKEN once deployed, or POST /api/crawl is open to anyone who finds it.
• The crawl button awaits the full run (~35s, mostly the polite request delay). On a serverless host raise the function timeout, or rely on the scheduled path and treat the button as a local convenience.

Commands

Command	Purpose
npm run dev	Start the dashboard
npm run crawl	Crawl every active site
npm run crawl -- --brand joola --country US	One site
npm run crawl:dry	Fetch and normalize, write nothing
npm run crawl -- --stages catalog,snapshot	Selected stages
npm run reviews	Reviews only, incremental
npm run reviews:full	Reviews only, re-reading every page
npm run reviews:probe	Validate a brand's review adapter against the live site — writes nothing, needs no database
npm run sites	List configured sites
npm run sync	Push src/config/ brands + storefronts into the database
npm run sync:dry	Show what sync would write
npm run schema	Regenerate supabase/schema.sql from supabase/migrations/
npm run report	Last runs, model overview, recent changes
npm test	Normalizer unit tests
npm run typecheck	tsc --noEmit

Exit codes: 0 clean · 1 partial (some stages failed, data written) · 2 error.

---

This is the beginning of the README. Read the rest at the original. Read the full README ↗