Overview
Follow The Money IL aggregates, normalizes, and analyzes Illinois campaign-finance data from both state and federal sources, making the flow of political money searchable and explorable in one place. It ingests Illinois State Board of Elections bulk filings alongside federal Schedule A/B/E data from the OpenFEC API, then layers on entity resolution, network analysis, and donor intelligence across election cycles. The platform tracks 6.4M+ itemized receipt rows, 530K+ D-2 filing totals, and 1.3M+ donor-analytics rows, and is deployed in production at followthemoneyil.com with scheduled syncs that keep federal and analytics data current.
Features
- Multi-source ingestion — Illinois State Board of Elections (ISBE) committee filings, D-2 reports, and itemized receipts/expenditures; federal candidate, contribution, disbursement, and independent-expenditure data via the OpenFEC API; plus IL Secretary of State lobbying, IRS 527 political-organization filings, City of Chicago open data, and OpenBook Comptroller contracts.
- Resilient scraping pipeline — Async Playwright scraping with ASP.NET postback/ViewState handling, resumable scrape state, configurable rate limiting, and exponential backoff.
- Bulk ETL — Chunked loading of large CSV/TXT bulk exports with amendment-safe aggregation, duplicate detection by source identifiers, and quarantining of malformed rows.
- Entity resolution & cross-matching — Jaccard-similarity matching across state and federal records, plus a cross-matching engine that links lobbying entities, 527 directors and organizations, donors, committees, and candidates.
- Analytics — Network/graph analysis, anomaly detection (large-contribution and monthly-spike flags), concentration metrics (HHI, Gini, top-N donor share), time-series aggregation, geographic donor rollups, and per-flag risk explainability.
- Interactive web app — Flask UI with sortable/filterable tables, a per-page time-scope bar (current/previous cycle, last 12 months, all time, custom range; the scope follows drill-down links), candidate/committee drilldowns, side-by-side compare mode, money-flow and network visualizations, a dark-money tracker, and unified cross-dataset person search, with CSV/JSON export throughout.
- Profiles, full lists and races — A candidate profile (
/candidates/<id>: all of a candidate's committees, most recently active first); full donor and payee lists per committee (/committees/<id>/donors, /committees/<id>/payees) and contributor / recipient lists per 527 organization (/527/<ein>/contributors, /527/<ein>/recipients), linked from each profile's top-N tables; every state race of a cycle (/analytics/state-races/); and an opt-in roll-up of serial committee re-registrations (/committees/?by=family) driven by admin-reviewed committee families (/admin/committee-families).
- Production-ready API surface — Optional API-key auth, per-key rate limiting, and Redis-backed route caching for heavy aggregate views.
Tech stack
- Python 3.12 — core language
- Flask 3 + Jinja2 — web framework and templating
- PostgreSQL 16 — primary database (local and production)
- Playwright — async browser automation for scraping
- OpenFEC API — federal campaign-finance data
- Click — CLI command framework
- NetworkX — graph/network analysis
- Redis — shared route cache (with in-process fallback)
- pandas / NumPy / Matplotlib — data processing and visualization
- Gunicorn — WSGI server in production
- pytest — test suite
How it works
Ingest Parse / normalize Store Serve
────── ───────────────── ───── ─────
ISBE bulk filings ┐ ┌── PostgreSQL ──┐
OpenFEC API ├─► chunked ETL + async ───┤ normalized ├─► Flask web UI
IL SOS lobbying │ Playwright scraping, │ tables + │ + JSON/CSV API
IRS 527 filings │ entity resolution, │ analytics │ (Redis-cached
Chicago open data │ cross-matching │ rollups │ heavy views)
OpenBook contracts ┘ └────────────────┘
- Ingest — Bulk ISBE exports are downloaded and staged; federal data is pulled from the OpenFEC API; lobbying, 527, Chicago, and OpenBook sources are scraped or pulled via their respective APIs.
- Parse / normalize — Raw records are cleaned, deduplicated (amendment-safe), and resolved into canonical donor, committee, and candidate entities.
- Store — Normalized records and precomputed analytics rollups are written to PostgreSQL.
- Serve — A Flask application exposes search, drilldowns, network visualizations, and a JSON/CSV API, with Redis caching for compute-heavy aggregate routes.
Scheduled jobs keep the data fresh: a recurring sync refreshes federal (FEC) contributions, rebuilds donor identities, and rebuilds analytics rollups.
Getting started
Prerequisites
- Python 3.12+
- PostgreSQL 16+
- An OpenFEC API key (optional, for federal data sync — free from api.open.fec.gov)
Install
git clone https://github.com/Doommen3/Illinois_campaign_finance.git
cd Illinois_campaign_finance
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
playwright install chromium
Configure
cp .env.example .env
# Set DATABASE_URL, FEC_API_KEY, and FLASK_SECRET_KEY in .env
createdb ilcf
export DATABASE_URL=postgresql://$(whoami)@localhost/ilcf
DATABASE_URL is required in all environments. (A legacy DATABASE_PATH SQLite setting exists for backward compatibility but is deprecated and unused for new setups.)
Initialize and run
# Create the database schema
python run.py init-db
# Sync federal (FEC) data for Illinois candidates
python run.py sync-fec-il-federal
python run.py rebuild-fec-donor-identities
# Build analytics rollups
python run.py refresh-analytics
# Start the development web server
python run.py runserver --port 5000
In production the app is served via Gunicorn against the WSGI entry point (wsgi.py). Run python run.py --help to see the full set of ingestion, scraping, and maintenance commands. Table shapes, join keys, and data quirks are documented in .claude/data-dictionary.md.
Privacy and removal requests
Contributor addresses come from public ISBE filings. Removal requests under the
Illinois Judicial Privacy Act (705 ILCS 90) and similar laws are honored by
scrubbing the address to ISBE's own Redaction Requested format while keeping
the contribution record. The public policy is at /privacy.
python run.py add-privacy-redaction --first Jane --last Doe --zip 60601 \
--requested-by "Requester" --request-date 2026-07-16 --statute "705 ILCS 90"
python run.py apply-privacy-redactions # re-apply after sync-prod-db; sunshine-import does this automatically
Runbook: docs/runbooks/privacy_takedown_response.md.
Committee families
Serial re-registrations of one political actor are grouped (never merged) into reviewable families; only accepted families appear on public pages.
python3 scripts/committee_families.py report --limit 30 # read-only dry run: proposed families and weak pairs
python3 scripts/committee_families.py seed # store new proposals as pending (idempotent); review at /admin/committee-families
Design: docs/committee_families_design.md.
Tests
pytest -q
A new windowed widget needs a row in tests/scope_widgets_registry.py and a probe in tests/test_scope_widgets.py; CI runs both.
Author
Devin Oommen — devinoommen.com · Oommen & Company
License
Released under the MIT License.
