FantasyScout
Live: https://fantasyscout.beronilla.rodeo · Spec: SPEC.md · Architecture: docs/architecture.md
FantasyScout is a WNBA-first basketball data platform with a fantasy front end. The fantasy app is what you use; the pipelines, models, tests and docs behind it are the platform, built in public.
sources ─► GitHub Actions ingest ─► Databricks lakehouse (dbt: bronze/silver/gold/ops)
─► quality gate ─► export to R2 ─► Cloudflare Worker ─► Astro app, DuckDB-WASM, Tableau, notebooks
What's inside
| Capability | How FantasyScout does it | Where to see it |
|---|
| Cloud lakehouse and shared data models | Databricks Free Edition (Unity Catalog, Delta), bronze/silver/gold, star schema | Architecture page, dbt docs |
| Pipelines from third-party APIs | Python ingestors for stats.wnba.com, stats.nba.com and ESPN, landing raw JSON in a volume | ingest/, run history page |
| Large datasets | Play-by-play and shot data for every game since 2018, incremental models, partitioning, Z-ordering | Performance notes, run timings |
| Validation, quality checks, alerts | dbt tests, basketball-logic checks, freshness, anomaly checks, alerts on failure | Public Data Health page |
| Reconciliation across sources | Stats API vs ESPN box scores per player-game, ID crosswalk, mismatch log | Reconciliation table on Data Health |
| Notebooks and dashboards on the same data | Databricks notebooks, Jupyter via SQL connector, Tableau Public, in-browser SQL console | Notebooks and Dashboards pages |
| Products for non-technical readers | Fantasy board, player scouting pages, weekly notes written for a coach or GM | The app itself |
| Documentation of pipelines and definitions | dbt docs site, data dictionary, runbook, ADRs | docs/ |
| Generative AI, kept honest | Deterministic numbers; an LLM only drafts notes from gold tables, schema-validated and cited | AI page, eval results |
| Tracking and event data | Play-by-play events, shot locations, tracking-derived stats where exposed | Shot charts, event models |
Status
Built phase by phase; each phase ends with something live and a PR that passes CI. Current: phase 0, skeleton. See docs/architecture.md and .sdlc/state.md.
Repo layout
ingest/ Python ingestors, one module per source; CLI: fs-ingest
export/ gold -> Parquet/JSON -> R2, manifest, latest pointer
ops/ results recorder and publish gate (ops.test_results, ops.publish_gate)
dbt/ dbt project: models/bronze|silver|gold|ops, tests/ (quality checks), seeds/, macros/, sample/
ai/ notes generator, prompts, evals/
notebooks/ 01_board_backtest, 02_streamer_signals, 03_game_context, 04_data_quality, 05_shooting; replay.py
web/ Astro app
worker/ Cloudflare Worker + wrangler.toml
infra/ self-hosted runner for the jobs that need a residential address
bundles/ Databricks Asset Bundle: jobs, schedules, targets
dashboards/ Tableau extract script + publish README
docs/ adr/, runbook.md, architecture.md, data-dictionary.md (generated)
.github/ ci.yml, deploy-web.yml (ingest.yml, build.yml, notes.yml arrive with their phases)
Run it locally
uv sync --group dbt && uv run pytest # Python
cd dbt && DBT_PROFILES_DIR=. uv run dbt build --target duckdb # lakehouse models on the fixture sample
cd ..
cd web && npm ci && npm run build && npm test # Astro app and Playwright
cd ../worker && npm ci && npm test # Worker against the real build
Full details, including the Windows note about workerd, are in CONTRIBUTING.md.
Data sources and terms
The league stats API and ESPN's public JSON are unofficial. Requests are throttled hard, raw payloads are never redistributed, and only derived tables are published. Non-commercial use.
