Product Finder
Crawls a brand's storefront, reconstructs its full product hierarchy — categories,
model families, generations, thicknesses, colourways, SKUs — and records price,
discount and stock history so trends accumulate over time.
Phase 1 target: JOOLA · United States · pickleball paddles.
Built to add countries (India, Vietnam, Australia) and brands as config, not code.
Verified against joola.com: 234 collections · 85 paddle products · 129 SKUs.
Quick start
1. Try the scraper with no database
Works immediately — no Supabase needed. Fetches the live catalogue and prints
what the normalizer resolved.
npm install
npm run crawl:dry
Expected output:
dry-run: product lines {"Perseus":19,"Hyperion":18,"Scorpeus":14,"Agassi":13,...}
dry-run: generations {"Pro IV":36,"(unresolved)":22,"Pro V":12,"3S":12,"Gen 1":2}
dry-run: skill tiers {"pro":58,"recreational":10,"performance":9,"unknown":7}
dry-run: thicknesses {"16mm":63,"14mm":36,"10mm":12,"(none)":8,"12mm":4}
(unresolved) generations are correct — Vision, Edge, Champion, Dash and Beacon
are genuinely generation-less lines.
2. Create the database
Create a Supabase project, then apply the schema by one of these routes.
A — script (no CLI login needed). Put the direct connection URI in .env as
SUPABASE_DB_URL (Settings → Database → Connection string → URI), then:
npm run migrate:dry # list what would be applied
npm run migrate # apply; records each version in schema_migrations
Each migration runs in its own transaction, so a failure rolls back cleanly and
re-running is safe.
B — Supabase CLI. Requires being logged in as an account that owns the project:
supabase login
supabase link --project-ref <your-ref>
supabase db push
C — dashboard. Paste supabase/schema.sql (all migrations concatenated, in
order) into the SQL editor and run once.
Then set the API credentials in .env:
cp .env.example .env
# SUPABASE_URL = https://<ref>.supabase.co
# SUPABASE_SERVICE_ROLE_KEY = the SECRET key (sb_secret_... or legacy service_role)
The publishable/anon key will not work for ingest. RLS is enabled
deny-by-default in ..._rls_and_seed.sql, so only the service-role identity can
read or write. Keep the secret key out of git — .env is already ignored.
Confirm the seed landed:
npm run sites
# JOOLA US https://joola.com shopify 15 collections
3. First real crawl
npm run crawl
npm run report
The first run is the baseline — it writes snapshots but no change events,
because there is nothing to compare against. From the second run onward,
variant_events fills with price moves, discounts and stock changes.
4. Schedule it
The history only exists if this runs regularly. A missed day is a permanent hole.
Local (Windows Task Scheduler) — registration block is in the script header:
powershell -ExecutionPolicy Bypass -File scripts\crawl.ps1
CI (GitHub Actions) — .github/workflows/crawl.yml runs daily at 09:00 UTC.
Add SUPABASE_URL and SUPABASE_SERVICE_ROLE_KEY as repository secrets.
Preferred once the repo is pushed: it does not depend on your PC being on.
The dashboard
npm run dev # http://localhost:3000
npm run build && npm run start
| Page | Shows |
|---|
/ | KPIs, deepest discounts, recent movement, Pro tier by generation |
/catalogue | Every model, filterable by tier / generation / line |
/models/[id] | Versions, thicknesses, colourways, athletes, SKU table, price history |
/reviews | Every review, searchable — full text, star filter, complaints-only, verified-only, brand-replied-only, with the reply chain inline |
/discounts | Everything on promotion, deepest first |
/changes | The derived change log |
/pipeline | Run history, errors, and a Run crawl now button |
Pages are server components reading Supabase with the service-role key. The
browser never talks to Supabase directly, so RLS stays deny-by-default and the
secret key never reaches the client.
The price chart is hand-rolled SVG on purpose. Charting libraries interpolate
linearly, which would draw a smooth slide between two observations and imply
prices that never existed. observed_at is when we looked, so the only honest
render is a step.
Before management sees it
- The site has no authentication. Anyone with the URL sees your pricing
intelligence. Put it behind Vercel password protection, Cloudflare Access, or
an auth layer before sharing the link outside the team.
- Set
CRAWL_TRIGGER_TOKEN once deployed, or POST /api/crawl is open to
anyone who finds it.
- The crawl button awaits the full run (~35s, mostly the polite request delay).
On a serverless host raise the function timeout, or rely on the scheduled path
and treat the button as a local convenience.
Commands
| Command | Purpose |
|---|
npm run dev | Start the dashboard |
npm run crawl | Crawl every active site |
npm run crawl -- --brand joola --country US | One site |
npm run crawl:dry | Fetch and normalize, write nothing |
npm run crawl -- --stages catalog,snapshot | Selected stages |
npm run reviews | Reviews only, incremental |
npm run reviews:full | Reviews only, re-reading every page |
npm run reviews:probe | Validate a brand's review adapter against the live site — writes nothing, needs no database |
npm run sites | List configured sites |
npm run sync | Push src/config/ brands + storefronts into the database |
npm run sync:dry | Show what sync would write |
npm run schema | Regenerate supabase/schema.sql from supabase/migrations/ |
npm run report | Last runs, model overview, recent changes |
npm test | Normalizer unit tests |
npm run typecheck | tsc --noEmit |
Exit codes: 0 clean · 1 partial (some stages failed, data written) · 2 error.
以上为 README 的开头部分。其余内容请阅读原文。 查看完整 README ↗
