sc-open-quests
How many StreetComplete quests are open in each Canadian city over 50,000 people (and in
Berlin, Hamburg and Bremen), by quest type, and how that changes day to day. Each number is what a StreetComplete user standing in that city
would see, with the app's default settings, on a pinned app version. Live site:
https://skfd.github.io/sc-open-quests/ (GitHub Pages from docs/).
This project is read-only: it never edits OpenStreetMap.
How it works
- Quest definitions come from the app's own build.
tools/sc-harness dumps every quest's resolved
element filter from StreetComplete into data/quests/v<SC>.json. Nothing is hand-written.
- The engine (
engine/, Kotlin on the JVM, running StreetComplete's own filter code) reads each
region's Geofabrik extract (config/regions.json: the provinces, the German city-states) and counts
each quest per city polygon (config/cities.json, 115 municipalities). It writes one snapshot per
extract to the archive, data/archive/<region>/snapshots/<asOfDate>.json (not committed).
run.py history rebuilds the per-city time series in docs/data/history/, the city index
docs/data/cities.json and the per-quest comparison docs/data/compare/ from all snapshots, and
publishes each region's newest snapshot in docs/data/<region>/snapshots/.- The site (
docs/index.html, no build step) reads that JSON.
Quests the app shows but this site can't count yet are listed on the page, greyed, with their status.
Quests that reopen only because a check date aged ("needs resurvey") are counted apart from ones that
still need mapping. See DESIGN.md for the method and DATA-FORMAT.md for
the JSON.
Setup
- Python 3.11+ (standard library only, no venv needed).
- JDK 21 for the engine.
run.py uses Android Studio's bundled JBR
(C:\Program Files\Android\Android Studio\jbr) when it exists, otherwise your JAVA_HOME.
count builds the engine (engine\gradlew.bat installDist) when it is missing or older than its
sources.
- About 40 GB of free RAM for a full Ontario pass (53 cities; the engine's heap is 40 GB).
- Extracts come from the shared store
~/Code/.osm-data, managed by ~/Code/osm-extracts
(override with OSM_DATA_DIR / OSM_EXTRACTS_TOOL). This project never downloads a PBF itself.
Usage
python run.py download # osm-extracts fetch <extract> once per distinct extract (it skips an unchanged one)
python run.py count # engine on each region's newest extract -> data/archive/<region>/snapshots/ (skipped if counted)
python run.py backfill # count every dated extract still in the store
python run.py history # archive -> docs/data/history/<city>.json, cities.json, compare/<quest>.json, newest snapshots
python run.py update # all three, then commit "data <asOfDate>" and push (--no-push: commit only)
python run.py planet # monthly: osm-extracts fetch planet-history, osm-extracts recut (every clip), the
# months the regions on clips lack, the "cadence": "weekly" regions -> archive,
# history, commit "data planet <date>", push (--no-push). `weekly` is its old name
python tools/backfill_history.py months --region europe-west --region europe-central --region europe-east --region europe-msk [--step 12] # a clip's regions by hand, sharing each slice
python tools/clip_polygon.py europe # after adding a city to a clip's region: its polygon and .poly files
update is the scheduled entry point; planet the monthly one. A region counted from an
osm-extracts clip ("extract": "europe": the planet inside a polygon, kept current by the daily
diffs) goes through update like a Geofabrik one; planet re-cuts the clip and adds its monthly points. A second run on the same extract downloads nothing, counts
nothing and commits nothing. count --force recounts.
Tests: python -m unittest.
To preview the site, serve the repo root (python -m http.server) and open /docs/. Before the
engine has published docs/data/, point it at the fixture:
/docs/?data=../tests/fixtures/data (regenerate with python tests/fixtures/make_site_fixture.py).
Scheduling (Windows)
Two PowerShell scripts manage a Task Scheduler entry. Run them as Administrator.
.\schedule-add.ps1 # daily at 13:15, appends to logs\scheduler.log
.\schedule-remove.ps1
The task is named kk-SCOpenQuests. It starts when available if the machine was off at 13:15 and
is stopped after two hours.
.\schedule-planet-add.ps1 # run.py planet on the first Monday of each month at 07:30, appends to logs\planet.log
.\schedule-planet-remove.ps1
kk-SCOpenQuests-planet starts when available and is stopped after fourteen hours; registering it
removes the old kk-SCOpenQuests-weekly. It never holds the pipeline lock across the daily 13:15 run
(DESIGN.md, "Weekly planet job"). It runs monthly because each run rewrites the 160 GB history file
(DESIGN.md, "Planet job: monthly while nothing counts from it").
Licence and attribution
- Code: GPL-3.0. The engine links StreetComplete's source (GPL-3.0), so the whole project
is GPL-3.0.
- Map data: © OpenStreetMap contributors, ODbL 1.0.
Extract by Geofabrik.
- City boundaries: from the OpenStreetMap relations listed in
config/cities.json.
