◆ TechPulse

An AI-curated tech news monitor. TechPulse scrapes Hacker News on a schedule with a headless browser, uses Claude to classify, score and summarise every story against your interests, stores everything in PostgreSQL, and sends a Slack alert only when something matters. A FastAPI back end serves a REST API and a live dashboard.

Built as a compact portfolio project covering browser automation, LLM integration, API design, data modelling, testing and containerised delivery.

🎬 Demo video (4:33): docs/techpulse-demo.mp4 (regenerate with python demo/record_demo.py)

---

Architecture

flowchart LR
    S[APScheduler<br/>every 30 min] --> P
    A[POST /api/runs] --> P
    subgraph P[Pipeline · one run at a time]
      direction LR
      SC[Scrape<br/>Playwright or httpx + retries] --> ST[Upsert<br/>SQLAlchemy]
      ST --> EN[Enrich<br/>Claude structured outputs]
      EN --> AL[Alert<br/>Slack webhook]
    end
    ST --- DB[(PostgreSQL)]
    DB --- API[FastAPI REST API]
    API --- UI[Dashboard]

Stage	What happens	Tech
Schedule	Interval trigger, or on demand via API. An asyncio.Lock prevents overlapping runs.	APScheduler
Scrape	Headless browser loads N pages and extracts rank, title, URL, points, author and comments in one in-page JS pass. Images, fonts and CSS are blocked. Retries use exponential backoff.	Playwright, tenacity
Store	Idempotent upsert: new stories are inserted, and existing ones get fresh points and comments.	SQLAlchemy 2, PostgreSQL / SQLite
Enrich	Batches of 30 un-enriched stories are sent to Claude. The response is parsed directly into Pydantic models (category enum, relevance 1–10, summary, tags).	Anthropic SDK, structured outputs
Alert	Stories at or above ALERT_THRESHOLD that haven't been alerted are posted to Slack, exactly once.	httpx
Audit	Every run is recorded with its trigger, status, counts, enricher, duration and error.	runs table

Reliability built in

• Failure isolation: a failed run is recorded and shown in the dashboard, and never crashes the scheduler.
• Self-healing: if enrichment fails, stories stay relevance IS NULL and are picked up on the next run.
• No duplicate alerts: an alerted flag is set when an alert is sent.
• Offline mode: without ANTHROPIC_API_KEY a deterministic keyword enricher takes over. With SCRAPER_SOURCE=fixture, the real Playwright extraction runs against a saved page, which CI and demos use.

---

Quick start

Docker (app + PostgreSQL)

cp .env.example .env          # optional: add ANTHROPIC_API_KEY, SLACK_WEBHOOK_URL
docker compose up --build

Open http://localhost:8000 for the dashboard, or http://localhost:8000/docs for the API docs.

Local (SQLite)

python -m venv .venv && source .venv/bin/activate     # Windows: .venv\Scripts\activate
pip install -e ".[dev,browser]"
python -m playwright install chromium                 # or set BROWSER_CHANNEL=chrome
cp .env.example .env

techpulse run-once        # one pipeline pass, prints a summary
techpulse serve           # API + dashboard + scheduler on :8000

Vercel (serverless)

The same code runs as a Vercel Function. When VERCEL=1 is set, the app adapts automatically:

Concern	Docker / local	Vercel
Scraper	Playwright (headless Chromium)	httpx + HTML parser. Same output; both are tested against the same fixture.
Scheduling	In-process APScheduler	Vercel Cron (daily on Hobby), plus an optional GitHub Actions workflow every 2 h, both calling GET /api/cron with Authorization: Bearer $CRON_SECRET
"Run now"	Background task (202)	Runs inline, with MANUAL_RUN_COOLDOWN_SECONDS to prevent abuse
Database	SQLite / Compose Postgres	Neon Postgres via DATABASE_URL (falls back to per-instance /tmp SQLite)

1. Import the GitHub repo at vercel.com/new (no build settings needed: pyproject.toml sets [tool.vercel] entrypoint).
2. Add a Postgres database: Storage → Create → Neon. This injects DATABASE_URL; postgres:// URLs are normalised automatically.
3. Set environment variables: CRON_SECRET (random, ≥16 chars), MANUAL_RUN_COOLDOWN_SECONDS=300, and optionally ANTHROPIC_API_KEY, SLACK_WEBHOOK_URL, SCRAPER_PAGES=2, SCHEDULE_LABEL="Vercel Cron · daily".
4. Optional, for more frequent runs: add the repo secrets TECHPULSE_URL and CRON_SECRET to enable .github/workflows/scheduled-run.yml.

Configuration

All settings come from environment variables or .env, and Pydantic validates them.

This is the beginning of the README. Read the rest at the original. Read the full README ↗