Concert Crawl

A concert calendar for Denver that crawls venue websites daily to produce an accurate, filterable, searchable event listing.

Architecture

flowchart TD
    subgraph Client
        Browser["Browser / PWA"]
    end

    subgraph Vercel
        Web["apps/web\nNext.js 15 (App Router)\nVercel"]
    end

    subgraph Hosted
        API["apps/api\nHono REST\nVercel (concert-crawl-api)"]
        CrawlerTrigger["apps/crawler (trigger)\nGitHub Actions, on demand"]
        CrawlerDaily["apps/crawler (daily)\nGitHub Actions (09:07 UTC)"]
    end

    subgraph Neon
        DB[("Neon Postgres\nbranches: production, staging\nDrizzle ORM")]
    end

    subgraph Packages
        CrawlerCore["packages/crawler-core\nPlaywright + extraction"]
        PricePipeline["packages/price-pipeline\nPrice fetch + submission"]
        DBPkg["packages/db\nSchema + migrations"]
        Types["packages/types\nShared TS types"]
        Logger["packages/logger\nPino"]
    end

    subgraph External
        Clerk["Clerk\nAuth + OAuth"]
        Anthropic["Anthropic\nClaude Haiku\n(LLM extraction)"]
        Ticketmaster["Ticketmaster API\nPrice data"]
        Stripe["Stripe\nPayments"]
        Resend["Resend\nEmail"]
        Sentry["Sentry\nErrors + Uptime + Crons"]
    end

    Browser -->|HTTPS| Web
    Web -->|REST /v1/...| API
    API --> DB
    CrawlerTrigger --> CrawlerCore
    CrawlerTrigger --> DB
    CrawlerDaily --> CrawlerCore
    CrawlerDaily --> PricePipeline
    CrawlerDaily --> DB
    CrawlerCore --> Anthropic
    PricePipeline --> Ticketmaster
    API --> Clerk
    Web --> Clerk
    API --> Stripe
    API --> Resend
    API --> Sentry
    Web --> Sentry
    CrawlerTrigger --> Sentry
    CrawlerDaily --> Sentry

Monorepo Structure

apps/
  api/        Hono REST API (/v1/...)
  web/        Next.js 15 (App Router) — UI + admin
  crawler/    Playwright + Claude Haiku extraction (GitHub Actions workflows)
packages/
  config/         Shared ESLint, TypeScript, Tailwind configs
  crawler-core/   Playwright browser + extraction logic
  db/             Drizzle ORM schema, migrations, seed
  logger/         Shared Pino logger
  price-pipeline/ Price fetch + community submission pipeline
  types/          Shared TypeScript types

Prerequisites

• Node.js >= 20
• pnpm >= 10 (npm install -g pnpm)
• OrbStack or Docker Desktop

Getting Started

# 1. Start local Postgres
docker compose up -d

# 2. Install dependencies
pnpm install

# 3. Set up environment variables
cp apps/api/.env.example apps/api/.env.local
cp apps/web/.env.example apps/web/.env.local
cp apps/crawler/.env.example apps/crawler/.env.local

# 4. Run migrations and seed the database
pnpm db:migrate
pnpm db:seed

# 5. Start all apps
pnpm dev

Service	URL
Web	http://localhost:3000
API	http://localhost:3001
Postgres	localhost:5433

Local Development — Seeding with Staging Data

For a prod-like local environment, you can sync real venue and event data from staging into your local Postgres, then layer on representative synthetic user data (profiles, watching, price submissions, feedback, etc.).

One-time setup (run after pnpm db:migrate):

# Wipe local DB, re-migrate, then apply staging snapshot + seed
docker compose down -v && docker compose up -d
pnpm db:migrate
pnpm db:sync-local

pnpm db:sync-local applies scripts/staging-snapshot.sql (real venue + event data from staging) then runs pnpm db:seed (synthetic users, roles, watching, price submissions, feedback).

The script is fully idempotent — safe to re-run after a fresh docker compose down -v && docker compose up -d.

Synthetic users seeded:

Handle	Clerk user ID	Role
hugh	user_3CoSI01LtxRAvremTpL0yDfuEJ6	admin
jess_r	user_fake_jess_0001	user
marco_d	user_fake_marco_0002	user
amy_k	user_fake_amy_0003	sub_admin
ghost	user_fake_ghost_0004	user

Common Commands

pnpm dev          # Start all apps in watch mode
pnpm build        # Build all apps and packages
pnpm test         # Run all tests
pnpm typecheck    # TypeScript check across all packages
pnpm lint         # ESLint across all packages
pnpm db:generate  # Generate a new migration from schema changes
pnpm db:migrate   # Apply pending migrations
pnpm db:seed      # Seed local database with Denver venues

Tech Stack

Concern	Choice
Monorepo	Turborepo + pnpm workspaces
API	Hono 4
Web	Next.js 15 (App Router)
Auth	Clerk (separate Production + Staging apps)
Database	Neon Postgres 16 + Drizzle ORM
Crawler	Playwright + Claude Haiku (LLM fallback)
Styling	Tailwind CSS + shadcn/ui
Testing	Vitest (unit/integration), Playwright (E2E)
Observability	Sentry (errors, uptime, crons)
CI	GitHub Actions
Web hosting	Vercel (concertcrawl.app, preview deploys per PR)
API	Vercel (concert-crawl-api, api.concertcrawl.app)
Crawler runs	GitHub Actions (crawler-daily.yml, crawler-triggers.yml)
DNS	Cloudflare (DNS only, no proxy)

This is the beginning of the README. Read the rest at the original. Read the full README ↗