Movie Words - moviewords
Live: https://moviewords.org — explore the words spoken in
two corpora: 34,556 English-original films (default) and 64,046 films of
every original language (translated subtitles, clearly labeled): what any
movie actually says, how words rise and fall across decades, and what makes a
film, a decade, or a genre sound like itself.
Every film's dialogue (from the OPUS OpenSubtitles corpus,
subtitles by OpenSubtitles.org) is reduced to a
bag of words and compared against the whole corpus with log-odds — which is how
you learn that The Godfather Part II's signature words are "corleone, fredo,
roth, michael, vito", Horror's are "help, god, please, her, house", and the film
that says "dude" the most is, of course, The Big Lebowski (120×).
Only derived word counts are published — no subtitle text is redistributed.
What's here
| Path | What it is |
|---|
pipeline/ | Python batch pipeline: OPUS + IMDb + TMDB → the published dataset. Runbook in pipeline/README.md |
app/ | The website: Vite + React + Tailwind + DuckDB-WASM, hosted on Cloudflare Pages. Dev guide in app/README.md |
docs/ARCHITECTURE.md | Start here to understand or reproduce the system — architecture, methodology, dataset contract, decisions, toolchain |
docs/FAQ.md | Methodology, limitations, and licensing questions — subtitles vs scripts, corpus bias, copyright position, poster policy |
docs/DATA-QUALITY.md | How bad subtitle files were found and handled — wrong films, commentary tracks, machine-translated and auto-captioned subtitles, and why old films "said" words they never said |
docs/superpowers/ | Original design spec and implementation plan |
docs/sessions/ | Session records (how this was actually built, with gotchas) |
Architecture in one paragraph
A batch pipeline (Python/DuckDB, fully resumable via per-movie caches) turns the
34GB OPUS corpus into ~2GB of Parquet + JSON + posters on a public Cloudflare R2
bucket. The site is a static SPA on Cloudflare Pages that reads pre-baked JSON
for hot paths and runs real SQL in the browser (DuckDB-WASM over HTTP range
requests) for everything interactive. There are no servers and hosting is free.
Details, decisions, and performance lessons: docs/ARCHITECTURE.md.
Reproducing from scratch
- Build the dataset —
pipeline/README.md. Needs
~60GB disk, a free TMDB API key
(TMDB_API_KEY or TMDB_API_TOKEN), and a few hours (we used a throwaway
GCP e2-highmem-4; a laptop works too). A 610-film starter dataset needing
neither is one command: uv run python scripts/build_demo_dataset.py.
- Host the data — any static host with range-request + CORS support; we
use an R2 public bucket (
pipeline/scripts/upload_r2.sh).
- Run the app —
cd app && npm install && npm run dev, pointing
VITE_DATA_BASE at your data host. Deploy anywhere static
(wrangler pages deploy dist --project-name moviewords).
Download the data
The full published dataset is five Parquet files per corpus on a public bucket
(the all-films corpus under all/ adds a sixth, word_year_lang.parquet) —
download them, or point DuckDB straight at the URLs. Fuller guide with more
example queries: docs/DATA.md.
| File | Size | Contents |
|---|
| movies.parquet | 0.8 MB | one row per film: title, year, genres, rating, word totals |
| words_by_movie/data.parquet | 88 MB | (imdb_id, word, count), sorted by film |
| words_by_word/data.parquet | 93 MB | same rows, sorted by word |
| word_year.parquet | 6 MB | (word, year, count, movie_count) for trends |
| word_meta.parquet | 3.5 MB | per-word commonness (zipf), part of speech, distinctiveness |
duckdb -c "SELECT title, year, words_per_minute FROM 'https://data.moviewords.org/movies.parquet' ORDER BY words_per_minute DESC LIMIT 10"
Full schema: the dataset contract.
Got an analysis idea we haven't thought of? Email
[email protected].
Data & licensing
Code: MIT (LICENSE). Published dataset: derived word counts,
CC BY-NC-SA 4.0 (LICENSE-DATA.md explains why - IMDb's
non-commercial dataset terms set the floor). Built from: OPUS OpenSubtitles
v2024 (Lison & Tiedemann, 2016) with subtitles from OpenSubtitles.org;
IMDb non-commercial datasets (information courtesy of IMDb, used with
permission); TMDB (metadata & posters — this product uses the TMDB API but
is not endorsed or certified by TMDB). Non-commercial project.
Questions about copyright, methodology, or bias - see the
FAQ. Rights holders: takedown requests to
[email protected] are honored promptly.
