gtfs-semantic-diff

日本語版: README.ja.md

gtfs-semantic-diff compares two versions of a GTFS feed and reports the differences as human-recognizable, semantic changes — a route discontinued, evening service cut, a stop renamed or relocated, departures shifted — instead of thousands of changed rows. Every raw field-level difference is structurally accounted for: it either becomes evidence for a change event or is reported as an unexplained residual, and the coverage ratio is measured on every run.

Try it: diff.gtfs.jp — upload any two GTFS zips, free, no sign-up. The site and reports are bilingual (English / Japanese, following your browser language).

Features

• Explanation ledger: level 0 enumerates every raw difference (all files, all fields). Each one must either back a change event as evidence or be reported as a residual — so "did we miss something?" has a measurable answer (explained_ratio, typically 1.0 on our verification feeds).
• ID-independent matching: even when stop_id / route_id / trip_id are reassigned between versions, entities are re-identified from content — names, coordinates, stop sequences, times. The resulting old↔new ID mapping is itself an output (mapping.json), usable as a backbone for longitudinal analysis across feed versions.
• Reports in cognitive units: a self-contained HTML report in four parts (whole feed / stops / per route / everything else). Route pages show the old and new timetables merged in published-timetable format, with added, removed and retimed trips marked.
• Verification view: every raw difference can be traced from GTFS file/row/value to the event that explains it to where it is displayed (three-layer traceability).
• Deterministic, rule-based: no ML or LLM in detection or classification. All thresholds live in config/default.toml. Same input, same output.
• JSON as the stable interface: the core is a pure function from two snapshots to a ChangeEvent JSON stream; HTML, Markdown, the web service, the JSON API and the MCP server are all consumers of it.
• Color-blind aware: bold, symbols, line styles and numbers are the primary channel in every output; color only reinforces.

Machine-readable outputs and AI integration

The hosted service exposes every layer of a comparison result:

Layer	URL pattern	Content
Digest	r/{pair}.digest.en.md (English) / .digest.md (Japanese) / .digest.json	Summary for LLMs and humans — numbers identical in both languages
Route detail	r/{pair}.routes.digest.json	Per-route changed trips (old/new trip_ids), counts by time band
ID mapping	r/{pair}.mapping.json	Adopted old↔new correspondences for stops / routes / trips
Full events	see r/{pair}/index.json	All 44 event types with evidence, plus raw diffs

• MCP server: https://diff.gtfs.jp/mcp — register it as a connector in Claude or ChatGPT (no auth) and query comparisons conversationally, including running new comparisons.
• Developer guide: diff.gtfs.jp/developers.html · API reference: diff.gtfs.jp/docs/ · llms.txt

Install

Python 3.11+ (developed on 3.14).

uv venv .venv.nosync --python 3.14   # .nosync needed if the repo is under iCloud sync (see note)
ln -s .venv.nosync .venv
uv pip install -e '.[dev]' --python .venv.nosync/bin/python

Note (macOS + iCloud): if the repository lives under an iCloud-synced directory (e.g. ~/Documents), a venv at .venv breaks: iCloud keeps restoring a hidden flag on .pth files in site-packages, and Python 3.14 ignores hidden .pth files. Directories named *.nosync are excluded from iCloud sync, hence the layout above.

Usage

# Compare two local GTFS zips (older one first) into a self-contained HTML report
gtfs-semantic-diff compare old.zip new.zip --html report.html

# Also emit ChangeEvent JSON / Markdown / raw diffs
gtfs-semantic-diff compare old.zip new.zip \
    -o events.json --report report.md --rawdiffs rawdiffs.json

# Lightweight HTML (same "core" bundle as the web service) or app+data split output
gtfs-semantic-diff compare old.zip new.zip --html-lite lite.html --html-dir out/

# AI-oriented digest (Markdown / JSON, numbers guaranteed identical to the report)
gtfs-semantic-diff compare old.zip new.zip --digest digest.md --digest-json digest.json

# Japanese feeds: fetch generations directly from gtfs-data.jp
gtfs-semantic-diff fetch --org nagai-unyu --feed Nagaibus
gtfs-semantic-diff compare --org chitetsu --feed chitetsubus \
    --old prev_2 --new prev_1 --html report.html

# Run only the L1 identity stage and inspect match confidence
gtfs-semantic-diff identity --org chitetsu --feed chitetsubus

• --config swaps the threshold TOML (default: config/default.toml).
• The web service (infra/, AWS CDK; S3 + CloudFront + Lambda) powers diff.gtfs.jp and can be deployed to your own AWS account.

How detection works

Events fall into 6 groups (routes, stop patterns, trips/times, stops, calendars, fares/metadata), 44 types in total. The exhaustive detection specification is docs/spec/detection.md (Japanese): L0 enumeration rules, L1 identity scoring, and per-rule detection conditions, evidence consumption and thresholds.

zip ×2 | gtfs-data.jp API
  → load/      normalized loading (day_type normalization, all .txt kept as strings)
  → diff0/     L0: exhaustive RawDiff enumeration (the ledger's denominator)
  → identity/  L1: cross-version identity — stop clusters / route families /
               stop patterns / route groups
  → events/    L2: rule cascade + residual accounting → ChangeEvent JSON (canonical)
  → report/    presentation model (cognitive units) → HTML / Markdown / digest

This is the beginning of the README. Read the rest at the original. Read the full README ↗