Card Scout CA (tcg-tracker)

Find the cheapest copy of any Yu-Gi-Oh! single across Canadian stores.

Live: https://card-scout-silk.vercel.app

Canadian players compare prices by opening five store tabs and searching each one. Card Scout CA does that for them: it checks every in-stock Yu-Gi-Oh! single at Canadian stores twice a day, matches the same printing across stores, and shows who's cheapest for each edition and condition, with price history and a link to buy on the store's own site.

Built by kiunga400-hub.

Highlights

• ~50,000 live listings from three stores, matched into one catalogue: ~2,760 cards are sold at both 401 Games and Hobbiesville, ~9,000 at both Trinity and 401.
• Every store's data, read from the source each store already uses. Two stores serve their catalogue from a search provider's API (Hotyon, Fast Simon); the third uses Shopify's product feed. No HTML scraping and no headless browsers.
• Shared card identity (CODE|rarity|art): titles like Ash Blossom & Joyous Spring [RA05-EN091] Ultra Rare, Ash Blossom - RA05-EN091 - Ultra Rare - 1st Edition and tag-only data all resolve to the same card. Anything that can't be identified is quarantined, never guessed.
• Trustworthy stock data: stores hide sold-out cards, so "out of stock" is inferred from absence, but only from a crawl that passed completeness checks. A partial or suspicious crawl changes nothing.
• Monitored like production: every run is checked (item counts, parse-problem rate, field null-rate jumps, HTTP errors) and failures go to Discord, one channel per store.
• Tested on real data: 216 tests, including adapter tests on saved full crawls of each store and end-to-end scrapy crawl runs against a mock site. CI on Python 3.12 and 3.14 with Postgres 16.
• Polite and permitted: each store gave permission to be listed; robots.txt obeyed, one request at a time, Retry-After honoured, a contact address in the User-Agent. Rate limits are respected, never worked around.

Stack: Python, Scrapy, PostgreSQL, FastAPI · Next.js 16, React, Tailwind CSS, shadcn/ui, Recharts · GitHub Actions.

How it runs (all free tiers, no PC needed)

flowchart LR
    subgraph GH["GitHub Actions"]
        C["crawl.yml<br/>07:15 & 19:15"]
        F["freshness.yml<br/>12:30"]
    end
    S["Store catalogues<br/>Hotyon · Fast Simon · Shopify"] --> C
    C -->|"verified runs"| DB[("Neon Postgres")]
    F --> DB
    C -. "alerts" .-> D["Discord"]
    F -. "alerts" .-> D
    DB --> API["FastAPI on Render"]
    API --> WEB["Next.js on Vercel"]
    WEB --> U["Visitors"]

Piece	Where	Notes
Crawls	GitHub Actions (.github/workflows/crawl.yml)	migrations once, then one job per store in parallel; a crashed or timed-out crawl alerts on Discord
Freshness check	GitHub Actions (freshness.yml)	alerts if a store has had no good crawl for 26 h, which catches crawls that never started
Database	Neon (Postgres 18, free plan, 1 GB)	~183 MB on 2026-10-27; CI tests on Postgres 16
API	Render (render.yaml, free plan)	sleeps after 15 idle minutes, so the first request after a quiet spell takes ~30-50 s
Website	Vercel (web/)	redeploys on every push to main

Secrets (NEON_DATABASE_URL, TCG_CONTACT_EMAIL, the Discord webhooks) live in GitHub Actions secrets and Render environment variables, never in the repo.

How it was built

I designed and directed this project and built it with an AI coding assistant (Claude Code). I chose the stores and got each one's permission to list their prices, set the requirements (reliability, loud failures, tests on real data), and made the product and engineering calls documented below: which data source to use for each store and why, how card identity works, when a crawl is trusted, how out-of-stock is decided, and how to handle stores that rate-limit bots.

Stores

Prices in CAD, in-stock Yu-Gi-Oh! singles only:

Store	Spider	Data source	Size (2026-10-26)
Trinity Hobby (trinityhobby.com)	trinity	Hotyon "USF" search API	~11,800 listings, 59 requests
401 Games (store.401games.ca, Toronto)	401games	Fast Simon search API	~35,000 listings, 141 requests
Hobbiesville (hobbiesville.com, Ottawa/Toronto)	hobbiesville	Shopify products.json	~4,200 listings, 18 requests, ~2 minutes at 5 s pacing
Fusion Gaming (fusiongamingonline.com)	fusion	Shopify products.json	~13,800 listings, 58 requests. Not scheduled: Shopify answered it with 429 at every pace we tried; the spider stays for manual runs

Cards line up across stores on the same card_key: Trinity–401 share ~9,000, 401–Fusion ~11,500, Trinity–Fusion ~4,200, and ~3,700 cards are at all three. Hobbiesville shares ~2,760 in-stock cards with 401 and ~1,830 with Trinity; ~1,380 are at all three scheduled stores (Trinity, 401, Hobbiesville).

Website

web/ is the front end (Next.js 16 App Router with cache components, Tailwind, shadcn/ui). It only talks to the API below, never to the database.

Page	What it shows
/	Search with suggestions as you type, deal of the day, stats, browse by set
/search?q=	Every printing matching a name or card number, cheapest price and number of stores
/cards/[cardKey]	Prices at every store per edition and condition, cheapest highlighted, links to buy, price history chart, other printings
/sets, /sets/[code]	Sets A–Z; a set's cards in number order with sort, rarity and store filters
/deals	Biggest savings for the same card, edition and condition between stores
/stores	Each store and when it was last checked

This is the beginning of the README. Read the rest at the original. Read the full README ↗