cinemetrics

Live: featheranalytics.dev/cinemetrics

Why this exists

A personal film analytics project that started as a spreadsheet and became an end-to-end pipeline: ingest, model, export, deploy, recommend. The interesting part is not the stack — it is what the data says when you stop lying to it about what it recorded.

What it does

• Cross-filtered dashboard: every chart responds to every filter in real time, and every view is a shareable URL (filters, stories, and per-chart deep links).
• Guided stories: one-tap findings (Spooktober, hidden gems, double features, franchise runs) that filter the charts and annotate what they show.
• Viewing habits over time: pace, seasonality, genre drift.
• Taste alignment with critics (Metascore, Rotten Tomatoes, IMDb).
• Rewatch patterns, rating changes, and franchise runs (TMDB collections rolled up into umbrella franchises like the MCU via a dbt macro).
• Recommendation engine: candidates ranked by cosine similarity to a taste vector built from my own ratings, with EN/Non-EN toggle, dashboard filter integration, and explainable results ("Why this film").

Data lessons

• Three-state liked — the 129 pre-Letterboxd rows never recorded liked or is_rewatch. Collapsing the unknowns into false understates the affection rate by 7.4 points. Always filter liked is not null first.
• Rewatch versus return — 214 rows are flagged as rewatches, but 88 of those are films whose first viewing predates the dataset. Counting return visits in the data gives 126 across 87 films. State which one a figure means.
• The factor of 20 is measured, not assumed — star_rating × 20 = rating_100 holds across the 683 rows that arrived carrying both values. Derived in stg_film_log.sql.
• Cut a contrast against its neighbors — this library changed character around mid-2022. Any "everything else" bucket is mostly the high-volume, low-rated early years, so a contrast against it measures the era, not the thing.

How it fits together

flowchart LR
  subgraph Sources
    letterboxd[Letterboxd RSS]
    tmdb[TMDB]
    omdb[OMDb]
  end
  subgraph Ingest
    update[update.py]
    fetch[fetch_candidates.py]
  end
  subgraph Model
    seeds[(seed CSVs)]
    dbt[dbt: staging → marts]
    db[(DuckDB)]
  end
  subgraph Publish
    export[export_web.py]
    train[train_embeddings.py]
    json[cinemetrics.json]
    r2[Cloudflare R2]
    site[GitHub Pages]
    drawer[Recommendation drawer]
  end

  letterboxd --> update
  tmdb --> update
  tmdb --> fetch
  omdb --> update
  omdb --> fetch
  update --> seeds
  fetch --> seeds
  seeds --> dbt
  dbt --> db
  db --> export
  db --> train
  export --> json
  train --> r2
  json --> site
  r2 --> drawer
  drawer --> site

  nightly{{Nightly Action}} -.-> update
  nightly -.-> fetch
  nightly -.-> dbt
  nightly -.-> export
  nightly -.-> train

• Seeds: committed CSVs — film_log.csv (watch history), film_enrichment.csv (rated films), candidate_enrichment.csv (recommendation pool from TMDB similar + popular).
• Staging: cleaned, typed views over the seeds (transform/models/staging/).
• Marts: dim_film, fct_watches, dim_candidate. The franchise_mapping() macro rolls TMDB collections up into umbrella franchises by collection, film, or director rules.
• ML pipeline: TF-IDF + multi-hot feature encoding → cosine similarity (recommend/encode.py). train_embeddings.py exports sparse vectors and R2 serves them; the browser () builds a rating-weighted taste vector and does the similarity math client-side. On a leave-one-out held-out test over liked films, the balanced ranker places the held-out film at median rank 5,071 with 2% in the top 100.

Layout

Path	Contents
recommend/	Python: ML pipeline (feature encoding, model, explainability)
ingest/	Python: TMDB + OMDb enrichment
transform/	dbt project (seeds → staging → marts)
scripts/	export, candidate fetch, training, R2 upload
tests/	pytest: encoding, model, ingest, taste eval
web/	Next.js dashboard + recommendation drawer
data/	movies.duckdb, ml/ (gitignored)

Setup

make setup                # install Python (uv) + Node dependencies
cp .env.example .env      # add your TMDB, OMDb, and R2 keys
make build                # full pipeline: dbt → export → train → web build
make dev                  # start Next.js dev server at localhost:3000

See .env.example for the full list of keys.

Commands

All targets are defined in the Makefile.

Command	What it does
make build	Full pipeline: dbt build → export JSON → train embeddings → web build
make dev	Start Next.js dev server
make test	Run all tests: ruff + eslint + dbt + vitest
make candidates	Fetch candidate films from TMDB (similar + popular)
make train	Train embeddings (skips if data unchanged)
make retrain	Force retrain regardless of data changes
make upload	Upload embeddings to Cloudflare R2
make update	Auto-update from Letterboxd RSS
make docs	Generate the dbt docs site into web/public/dbt/
make doc-stats	Refresh the docs' figures and lineage diagram from the built marts

Documentation

• docs/ARCHITECTURE.md — how the pipeline and infrastructure fit together, plus the data conventions that are easy to get wrong.
• Data model docs — generated dbt catalog, column descriptions, and lineage graph. The dashboard footer links here too.

Data sources

Letterboxd for the watch log (ingest/letterboxd.py), TMDB for genres/keywords/runtime/budget/revenue/similar films, OMDb for critic scores, box office, and cast. API clients live in ingest/http.py. Primary key: tmdb_id.

License

The code is MIT-licensed. The seed CSVs under transform/seeds/ are personal data shared for reproducibility, not relicensed.