Illinois Campaign Finance Tracker

Overview

Follow The Money IL aggregates, normalizes, and analyzes Illinois campaign-finance data from both state and federal sources, making the flow of political money searchable and explorable in one place. It ingests Illinois State Board of Elections bulk filings alongside federal Schedule A/B/E data from the OpenFEC API, then layers on entity resolution, network analysis, and donor intelligence across election cycles. The platform tracks 6.4M+ itemized receipt rows, 530K+ D-2 filing totals, and 1.3M+ donor-analytics rows, and is deployed in production at followthemoneyil.com with scheduled syncs that keep federal and analytics data current.

Features

• Multi-source ingestion — Illinois State Board of Elections (ISBE) committee filings, D-2 reports, and itemized receipts/expenditures; federal candidate, contribution, disbursement, and independent-expenditure data via the OpenFEC API; plus IL Secretary of State lobbying, IRS 527 political-organization filings, City of Chicago open data, and OpenBook Comptroller contracts.
• Resilient scraping pipeline — Async Playwright scraping with ASP.NET postback/ViewState handling, resumable scrape state, configurable rate limiting, and exponential backoff.
• Bulk ETL — Chunked loading of large CSV/TXT bulk exports with amendment-safe aggregation, duplicate detection by source identifiers, and quarantining of malformed rows.
• Entity resolution & cross-matching — Jaccard-similarity matching across state and federal records, plus a cross-matching engine that links lobbying entities, 527 directors and organizations, donors, committees, and candidates.
• Analytics — Network/graph analysis, anomaly detection (large-contribution and monthly-spike flags), concentration metrics (HHI, Gini, top-N donor share), time-series aggregation, geographic donor rollups, and per-flag risk explainability.
• Interactive web app — Flask UI with sortable/filterable tables, a per-page time-scope bar (current/previous cycle, last 12 months, all time, custom range; the scope follows drill-down links), candidate/committee drilldowns, side-by-side compare mode, money-flow and network visualizations, a dark-money tracker, and unified cross-dataset person search, with CSV/JSON export throughout.
• Profiles, full lists and races — A candidate profile (/candidates/<id>: all of a candidate's committees, most recently active first); full donor and payee lists per committee (/committees/<id>/donors, /committees/<id>/payees) and contributor / recipient lists per 527 organization (/527/<ein>/contributors, /527/<ein>/recipients), linked from each profile's top-N tables; every state race of a cycle (/analytics/state-races/); and an opt-in roll-up of serial committee re-registrations (/committees/?by=family) driven by admin-reviewed committee families (/admin/committee-families).
• Production-ready API surface — Optional API-key auth, per-key rate limiting, and Redis-backed route caching for heavy aggregate views.

Tech stack

• Python 3.12 — core language
• Flask 3 + Jinja2 — web framework and templating
• PostgreSQL 16 — primary database (local and production)
• Playwright — async browser automation for scraping
• OpenFEC API — federal campaign-finance data
• Click — CLI command framework
• NetworkX — graph/network analysis
• Redis — shared route cache (with in-process fallback)
• pandas / NumPy / Matplotlib — data processing and visualization
• Gunicorn — WSGI server in production
• pytest — test suite

How it works

  Ingest                Parse / normalize           Store              Serve
  ──────                ─────────────────           ─────              ─────
  ISBE bulk filings  ┐                          ┌── PostgreSQL ──┐
  OpenFEC API        ├─► chunked ETL + async ───┤   normalized   ├─► Flask web UI
  IL SOS lobbying    │   Playwright scraping,    │   tables +     │   + JSON/CSV API
  IRS 527 filings    │   entity resolution,      │   analytics    │   (Redis-cached
  Chicago open data  │   cross-matching          │   rollups      │    heavy views)
  OpenBook contracts ┘                          └────────────────┘

1. Ingest — Bulk ISBE exports are downloaded and staged; federal data is pulled from the OpenFEC API; lobbying, 527, Chicago, and OpenBook sources are scraped or pulled via their respective APIs.
2. Parse / normalize — Raw records are cleaned, deduplicated (amendment-safe), and resolved into canonical donor, committee, and candidate entities.
3. Store — Normalized records and precomputed analytics rollups are written to PostgreSQL.
4. Serve — A Flask application exposes search, drilldowns, network visualizations, and a JSON/CSV API, with Redis caching for compute-heavy aggregate routes.

Scheduled jobs keep the data fresh: a recurring sync refreshes federal (FEC) contributions, rebuilds donor identities, and rebuilds analytics rollups.

Getting started

Prerequisites

• Python 3.12+
• PostgreSQL 16+
• An OpenFEC API key (optional, for federal data sync — free from api.open.fec.gov)

Install

git clone https://github.com/Doommen3/Illinois_campaign_finance.git
cd Illinois_campaign_finance

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
playwright install chromium

Configure

cp .env.example .env
# Set DATABASE_URL, FEC_API_KEY, and FLASK_SECRET_KEY in .env

createdb ilcf
export DATABASE_URL=postgresql://$(whoami)@localhost/ilcf

DATABASE_URL is required in all environments. (A legacy DATABASE_PATH SQLite setting exists for backward compatibility but is deprecated and unused for new setups.)

Initialize and run

# Create the database schema
python run.py init-db

# Sync federal (FEC) data for Illinois candidates
python run.py sync-fec-il-federal
python run.py rebuild-fec-donor-identities

# Build analytics rollups
python run.py refresh-analytics

# Start the development web server
python run.py runserver --port 5000

In production the app is served via Gunicorn against the WSGI entry point (wsgi.py). Run python run.py --help to see the full set of ingestion, scraping, and maintenance commands. Table shapes, join keys, and data quirks are documented in .claude/data-dictionary.md.

Privacy and removal requests

Contributor addresses come from public ISBE filings. Removal requests under the Illinois Judicial Privacy Act (705 ILCS 90) and similar laws are honored by scrubbing the address to ISBE's own Redaction Requested format while keeping the contribution record. The public policy is at /privacy.

python run.py add-privacy-redaction --first Jane --last Doe --zip 60601 \
  --requested-by "Requester" --request-date 2026-07-16 --statute "705 ILCS 90"
python run.py apply-privacy-redactions   # re-apply after sync-prod-db; sunshine-import does this automatically

Runbook: docs/runbooks/privacy_takedown_response.md.

Committee families

Serial re-registrations of one political actor are grouped (never merged) into reviewable families; only accepted families appear on public pages.

python3 scripts/committee_families.py report --limit 30   # read-only dry run: proposed families and weak pairs
python3 scripts/committee_families.py seed                # store new proposals as pending (idempotent); review at /admin/committee-families

Design: docs/committee_families_design.md.

Tests

pytest -q

A new windowed widget needs a row in tests/scope_widgets_registry.py and a probe in tests/test_scope_widgets.py; CI runs both.

Author

Devin Oommen — devinoommen.com · Oommen & Company

License

Released under the MIT License.