Boston Urban Hazard Maps

Interactive heatmaps of publicly available data from Boston's Analyze Boston open data portal (311 Service Requests).

We map whatever geolocated 311 data is available and interesting. Right now that's three datasets:

• Sharps collection requests — reports of discarded needles/syringes for safe pickup
• Encampment reports — 311 requests filed under the "Quality of Life" category (2023–present; the dedicated type ended May 27, 2026)
• Human waste (beta) — street-cleaning requests classified by text as human waste

These are independent datasets. They come from the same 311 system but are unrelated complaint types. We display them on the same map because it's useful to see where the city is responding to different kinds of issues — not because we're claiming any connection between them.

---

How it works

┌─────────────────────┐     ┌──────────────────────────┐     ┌─────────────────────┐     ┌──────────────────┐
│  data.boston.gov     │────>│  pipeline/ (daily cron)   │────>│  S3 bucket (Tigris)  │────>│  frontend/ (Astro │
│  CKAN + Open311      │     │  Python 3.12 · uv · spaCy │     │  precomputed JSON    │     │  SSR + React map) │
└─────────────────────┘     └──────────────────────────┘     └─────────────────────┘     └──────────────────┘

Key details:

• Data source: Analyze Boston 311 Service Requests (CKAN), plus the city's Open311 feed scraped by services/open311-scraper/
• Sharps filter: TYPE in ("Needle Pickup", "Needle Clean-up", "Needle Cleanup"), 2015–present
• Encampment filter: TYPE = "Encampments" (2025+) plus tickets routed to the internal encampment queues (2023+); the dedicated type stopped appearing after May 27, 2026 (see docs/wiki/encampment-intake-ended-2026.md)
• Human waste (beta): TYPE = "Requests for Street Cleaning" classified with a spaCy model, 2024–present
• Pipeline: runs daily at 07:00 UTC on Railway, fetches the current year, computes stats, writes JSON to an S3-compatible bucket, and publishes a per-source health file (docs/wiki/source-health-runbook.md)
• Frontend: Astro SSR reads the bucket with a 5-minute cache; the client filters points by year/month in the browser
• Deployment: Railway (two services: pipeline cron + frontend). No GitHub Actions deploy.

---

Setup

Prerequisites

• Python 3.12 or 3.13 (spaCy has no 3.14 wheels yet; pipeline/.python-version pins 3.13)
• uv (Python package manager)
• pnpm (frontend package manager)
• lefthook (git hooks)
• Docker (optional, for a local MinIO bucket)

Install

git clone https://github.com/urban-hazards/urban-hazard-maps.git
cd urban-hazard-maps

# Pipeline
cd pipeline
uv sync
uv run python -m spacy download en_core_web_sm   # re-run after any `uv sync` that drops it
cp .env.example .env                              # points at local MinIO by default

# Frontend
cd ../frontend
pnpm install
cp .env.example .env

# Git hooks
cd ..
lefthook install

---

Usage

Run locally

# Local S3 (MinIO) — from the repo root; console at http://localhost:9001
docker compose up -d

# Run the pipeline once (from pipeline/)
uv run boston-pipeline                 # all datasets
uv run boston-pipeline -d needles      # one dataset
uv run boston-pipeline -d waste --force  # force re-fetch from CKAN
uv run boston-pipeline --verbose

# Start the frontend (from frontend/, port 4321)
pnpm dev

The frontend reads whatever the pipeline last wrote to the bucket. CARTO_BASEMAP_KEY (optional) enables the CARTO basemap; without it the map falls back to Esri's keyless tiles.

---

Development

Linting, type checking, tests

Pipeline (from pipeline/):

uv run ruff check src/ tests/
uv run ruff format src/ tests/
uv run mypy src/
uv run pytest

Frontend (from frontend/):

pnpm check    # astro check + biome check
pnpm test     # vitest
pnpm lint     # biome lint
pnpm format   # biome format

Git hooks (via lefthook) run ruff, mypy, and biome on commit. All changes go through PRs; .github/workflows/pr.yml must pass before merge.

---

Project Structure

urban-hazard-maps/
├── pipeline/                       # Daily Python cron job (Railway)
│   ├── src/pipeline/
│   │   ├── run.py                  # Orchestrator: fetch → clean → classify → compute → write
│   │   ├── cli.py                  # Typer CLI (`boston-pipeline`)
│   │   ├── config.py               # CKAN resource IDs, type filters, S3 env vars
│   │   ├── fetcher.py              # CKAN fetching (type + queue strategies)
│   │   ├── cleaner.py              # Record normalization & validation
│   │   ├── classifier.py           # spaCy human-waste classifier
│   │   ├── analytics.py            # Heatmap bins, neighborhoods, hourly/monthly stats
│   │   ├── health.py               # Per-source freshness → metadata/source_health.json
│   │   └── storage.py              # S3 read/write
│   ├── tests/
│   ├── Dockerfile
│   └── pyproject.toml
├── frontend/                       # Astro SSR site (Railway)
│   ├── src/
│   │   ├── pages/                  # Astro pages
│   │   ├── components/             # Astro components + React islands (HeatMap)
│   │   ├── lib/                    # bucket reader, types, freshness model
│   │   └── styles/
│   ├── Dockerfile
│   └── package.json
├── services/open311-scraper/       # Open311 day-file scraper (Railway)
├── docs/
│   ├── wiki/                       # What we know about Boston 311 (start at INDEX.md)
│   ├── design/                     # Design docs + review transcripts
│   └── architectural-decisions/
├── docker-compose.yml              # Local MinIO
├── CLAUDE.md                       # Project guide for AI assistants
└── lefthook.yml

---

Data Sources

All data comes from the City of Boston's Analyze Boston open data portal, published under the Open Data Commons PDDL license.

Dataset	311 Type	Available	Description
Sharps	Needle Pickup, Needle Clean-up	2015–present	Reports to the city's Mobile Sharps Collection Team for safe retrieval of discarded sharps in public spaces
Encampments	Encampments + internal encampment queues	2023–present (type stopped May 27, 2026)	311 reports filed under the "Quality of Life" category
Human Waste (beta)	Requests for Street Cleaning, NLP-classified	2024–present	Street-cleaning requests whose text describes human waste; see /methodology

These are separate complaint types within the same 311 system. People call 311 for all kinds of reasons — potholes, noise, graffiti, needles, encampments, etc. We picked these two because they have good geolocation data and are relevant to public health. Showing them on the same map is a convenience, not a claim that they're related.

As more useful 311 categories become available, we may add them.

---

License

MIT