pgx
PGX — Lightweight Postgres for AI agents. Written in Rust. Thousands of isolated databases in one runtime. 🐘
翻译准备中
尚未检测到更新。GitHub 发布版本后会显示在这里。
个人 GitHub 账号下公开仓库的所有者可以登录并验证所有权,自行管理介绍,无需管理员批准。
登录并生成验证文件,在 30 分钟内将其提交到此仓库的默认分支。其他分支或分叉仓库中的文件无效。然后在下方检查文件,即可无需管理员批准直接管理项目。请保留文件以维持管理权限。
如需登录,将前往登录页面。登录后请回到此处,再次点击所选的验证按钮。
在项目文件夹中运行 /nomorevibe verify,即可验证所有权并自行管理介绍。
一分钟完成验证正在加载完整 README。 以下为已保存的摘录。
This repository powers the public BEV Trajectory Gallery, an interactive dashboard that visualizes electric-vehicle transition dynamics for countries around the world. ▶ Open the live gallery: https://leraffl.github.io/LeRaffl-Gallery/#gallery The gallery is updated continuously and provides, for each country: • 📈 BEV transition trajectories (fitted logistic / Weibull-style curves) • 🔄 ICE↔BEV market-share transition charts • 🟫 TTM (trailing-twelve-month) market-split graphs • ⏱️ Transition-time curves (speed of change between thresholds)查看完整 README ↗
This repository powers the public BEV Trajectory Gallery, an interactive dashboard that visualizes electric-vehicle transition dynamics for countries around the world.
▶ Open the live gallery:
https://leraffl.github.io/LeRaffl-Gallery/#gallery
The gallery is updated continuously and provides, for each country:
In addition, the dashboard includes two analytical views:
All computations run purely client-side in the browser.
Many countries follow an S-curve when transitioning from internal-combustion engines to battery-electric vehicles.
This project aims to make these transitions comparable, transparent, and publicly accessible, using a consistent modelling approach across all markets.
Inputs include:
Outputs:
params.csv) containing all fitted model parametersbands/<slug>.json: confidence, prediction and tolerance intervals)manifest.json)Key files in the repository root:
| File | Purpose |
|---|---|
| index.html | The full interactive UI, single file, no build step. Four primary nav entries — Charts / Map / Rankings / Tools — with a sub-nav, over 14 sections (Gallery, World Map, Thresholds, Durations, Time Interval, Builder, Compare, Raw Data, Fleet, Data freshness, Submit Data, About, FAQ, Feedback). Runs 100% in the browser. |
| manifest.json | Auto-generated list of all available charts, used by the Gallery to display images. |
| build_manifest.R | Scans images/ and generates manifest.json. |
| params.csv | Contains the model parameters (v1, v2, t0, baseline year, last data month) for each market. Used for Thresholds & Durations. |
| bands/ | Generated per render by R/bands.R: one <slug>.json per series with the 95 % confidence, prediction and tolerance bands around the fitted curve, plus the 10/20/50/80/90 % crossing years with their confidence ranges. Frontend data only (the PNGs do not use it): drawn as the optional uncertainty band in Builder and Compare. What the bands mean and how to quote them: docs/architecture/44-uncertainty-bands.md. |
| footnotes.csv | Optional curated per-country/variant chart footnotes (country,variant,footnote). render_country.R appends the matching note as a second caption line on the PNGs (e.g. Canada Whole's pre-2017 passenger-cars-only scope). Separate from the CSVs' internal notes column. |
| log_params.R | Produces and updates the parameter table (params.csv). |
| images/ | Contains all exported PNG files structured as images/YYYY-MM/.... GitHub Pages serves them directly. |
| .github/workflows/build-manifest.yml | CI workflow ensuring that manifest.json always stays up to date. |
| docs/architecture/ | Architecture handbook — components, data objects, interfaces, flows, secrets, ops. Kept up to date with every PR that changes structure. Start at docs/architecture/README.md. |
The rows above are the ones that grew a long-form entry; they are not the
full set. Data is fetched automatically for 30 countries in total — the ones
without a row here are Albania, Austria, China, Colombia, Denmark, Finland,
Ireland, Italy, Luxembourg, Malaysia, Netherlands, Poland, Portugal, Singapore,
Spain and Sweden, plus the ~13-country ACEA cluster
(scripts/fetch_acea.py).
A related but separately-curated set of countries gets Vans / HDV /
Buses variants from a second, separate fetcher — scripts/fetch_acea_cv.py,
ACEA's Commercial Vehicle press release — since ACEA publishes cars and
commercial vehicles as two different reports, and which countries need which
report differs (Germany/France/Sweden have their own passenger-car source
but no national commercial-vehicle one, so they're in this roster and not
the fetch_acea.py one; Switzerland left it in 2026-10 for ASTRA's monthly
register counts; Greece stays in it although its passenger cars come from SEAA,
because SEAA has no fuel split for commercial vehicles; see
docs/architecture/38-source-acea-cv.md § 1a).
ACEA only publishes cumulative year-to-date
checkpoints (Q1/H1/Q1-Q3/full-year) for commercial vehicles, so this fetcher
reconstructs genuine Q1/Q2/Q3/Q4 rows from them, falling back to a
single yearly row only when no quarterly baseline exists yet for that
country/variant/year; see
docs/architecture/38-source-acea-cv.md
for how.
For the complete, always-current picture — every fetcher, its source, the
variants it writes and its schedule — see
docs/architecture/02-components.md § 2.7
and the cron table in
docs/architecture/08-deploy-ops.md § 8.11.
Each country also has a public source page under
sources/ explaining
where its numbers come from.
When opening index.html, the browser loads:
The landing section opens with a hand-rolled inline-SVG chart for your home market, picked from your browser's timezone — no IP lookup, no location prompt — and a search box that writes straight through to the gallery filter (press Enter to jump to the gallery).
Charts are generated locally using R scripts.
The workflow:
build_manifest.R scans the directory → produces manifest.jsonparams.csv is carrying relevant output parameters from each modelThe tables calculate thresholds and durations live.
Why isn't country X on the gallery? Some countries were investigated and deliberately left off because their freely-available data is incomplete (e.g. the dominant EV brand doesn't report), paywalled, login-walled, or because the market doesn't fit the new-registration / organic-transition model (e.g. Ethiopia's 100 %-EV import mandate over an ~85 %-used-import market). The rationale per country (Mexico, Ethiopia, … — plus Argentina and Colombia, which were shelved there first and have since been added from better sources) is documented in docs/architecture/14-data-source-gaps.md.
For each country, I model the BEV share of new car registrations over time. Empirically, this share does not behave like random noise. It behaves like a structured transition process: slow start, acceleration, eventual stabilization. The model is therefore not trying to “predict the future”, but to describe the structure of the transition as observed so far.
Most large-scale technology transitions follow a similar qualitative pattern:
This produces curves that are monotonic, bounded, and nonlinear. An S-shape is not assumed because it is pretty, it is assumed because it is the simplest structure that repeatedly matches real transitions. We've seen transitions between technologies in the past and this shape just fits.
A normal distribution can, in fact, be fitted to cumulative adoption data like this. The problem is not that it can't be used. The problem is structure.
A normal distribution is:
In other words: A normal distribution is just less suited.
(1) Asymmetry Real-world adoption curves are almost never symmetric. The Weibull allows early-heavy, late-heavy, or roughly symmetric transitions — all with the same functional form.
(2) Failure visibility A Weibull does not HAVE to produce an S-curve. It could yield other shapes as well. It doesn't produce other shapes in most cases though, simply because the data fits S-shapes best. If the data does not support a meaningful transition, the model can:
Example: Markets like Japan, which show no clear BEV transition, cause the model to break — exactly where it should. These examples are rare though. At time of writing I know of exactly 2 cases: Japan and Croatia
(3) Parametes The model uses two parameters. fewer → insufficient flexibility more → unstable estimation and overfitting Two parameters are a sweet spot, flexible enough to reflect reality, constrained enough to remain interpretable.
(4) Weights and calculation The basis for the fit was a basic Weibull distribution function of the form 1-exp(v[1]*x^v[2]). This function is being fitted to the data via a numerical OLS approach that introduces the market sizes as weights. These weights have turned out to be necessary in cases where countries have highly fluctuating market sizes. That is because BEV have proven to be much more stable in their absolute number of registrations than other fuel types, meaning that if in a certain month, there are much fewer newly registered cars, then this affects BEV much less than Petrol, Diesel or even Hybrids. As an aftereffect this means that with a relatively constant absolute number compared to this now shrunken total number, BEV share would produce an outlier that pulls the BEV share upwards more than it should. Weights correct this and make cases like Ireland much more stable. Convergence criteria are set to maxit = 100000 and reltol=10^-30.
v2 — transition shape Controls where acceleration happens. Intuitively: Does the market ramp up early, or only after a long hesitation phase?
t0 — time shift It's a horizontal shift so the curve aligns nicely with calendar time. Ignore this one.
Technically these are assumptions. However, would we not assume this we'd introduce either less stable models or subjective models (which goes against what I want this to be). However, both the 0% and the 100% seem to fit reality quite well. Would they not fit reality, models would eventually output nonsense. If real data contradicts these bounds, the model stops fitting and this breakdown is visible. So far, mature markets (e.g. Norway or Denmark) genuinely converge towards 100%, not 70% or 80%.
ICE and PHEV curves Since the BEV curves fits so well, I have decided to redo an analogue curve for newly registered ICE cars as well. I struggled with the definition of what to call an ICE in this definition, but ended up at the split BEV & PHEV & ICE, meaning ICE in this categorisation includes HEV, H2, GAS, PETROL, DIESEL, ETHANOL and all other fuel types that are not BEV or PHEV (this also means that EREV are treated as PHEV). I could have chosen to define a Hybrid category for PHEV+HEV and sometimes I do even use it due to lack of more granular data (Türkiye), but this BEV-PHEV-ICE split seems like the one that yields the best insight on these trajectory curves. As a result of producing ICE and BEV curves, the leftover PHEV part should now not only make up the remaining percentage points to get to 100% if summed up, but if models behave, should also fit against real world data, so can be used as a sort of verification. Thus the graph showing 3 curves is produced.
This is a statement about TODAY This deserves to be absolutely unambiguous:
This is not a forecast. This is not a prediction. This is not a statement about the future.
The curve represents a best-fit description of the transition as of today. If future data changes the trajectory also changes, which is why I update it regularly. The parameters change, the curve changes, the derived thresholds change.
Think of it as: “Given everything we know right now, what would the transition look like if things simply continued?”
Not: “What will happen?”
Charts and model outputs may be shared freely with attribution to @LeRaffl. Original data sources are subject to their respective licenses. Feel free to use these outputs for your own purposes, but please link me to whatever you do.
| scripts/fetch_brazil.py | Automated data ingestion for Brazil. Queries ANFAVEA's "Central de Dados" (WordPress admin-ajax.php, registrations by fuel for cars + light commercials) for the running year and upserts the monthly rows into data/Brazil.csv. Until the 2026-09 site relaunch it parsed the yearly xlsx instead. Full parsing logic & column mapping documented in the module docstring; offline regression test in scripts/test_fetch_brazil.py. |
| .github/workflows/fetch-brazil.yml | Runs fetch_brazil.py on the 10th of each month (08:50 UTC) and on manual dispatch. If data/Brazil.csv changes, the workflow commits it and triggers render-country.yml with country=Brazil. |
| scripts/fetch_chile.py | Automated data ingestion for Chile. Discovers the latest ANAC PDFs (Mercado Automotor + Cero y Bajas Emisiones) for the previous calendar month, parses TOTAL from the bar chart and BEV/PHEV/HEV from the summary table, and writes the new row to data/Chile.csv only when both PDFs are available. Self-throttles via the CSV's latest period. |
| .github/workflows/fetch-chile.yml | Runs fetch_chile.py daily from the 14th of each month (08:20 UTC) and on manual dispatch — most runs are no-ops until ANAC publishes both PDFs. If data/Chile.csv changes, the workflow commits it and triggers render-country.yml with country=Chile. |
| scripts/fetch_japan.py | Automated data ingestion for Japan. Discovers the latest JADA monthly "燃料別メーカー別登録台数(乗用車)" file on jada.or.jp/pages/342/ (XLSX preferred, PDF fallback), extracts the 乗用車計 (all-makes total) row for the target month — PETROL/HEV/PHEV/DIESEL/BEV/(FCV+その他)/TOTAL — and upserts into data/Japan.csv. Scope is 登録車 only (no 軽自動車). Self-throttles via the CSV's latest period; manual override via --xlsx-url/--pdf-url when the JADA host blocks the runner IP. |
| scripts/fetch_thailand.py | Automated data ingestion for Thailand. Logs into the TAI AIU member portal (aiu.thaiauto.or.th, cookie session — no bearer token) and reads its JSON API on taiapi.thaiauto.or.th:3000: resolves the AIU website_id via /websites, then one GET /veh_reg_fuel/report?period_mode=year&year=<Y>&type_code=ALL per year returns every vehicle-type category (row array under raw_rows). Exact-type_label match splits four variants from that one call: Whole = Passenger Car and Pickup Truck (pickups are bundled, not separable) → data/Thailand.csv; plus HDV (Truck), Buses (Bus), 3-Wheelers (Three Wheelers). Maps BEV/PHEV/HEV←units, ICE←icev_units, OTHERS←other+not_specific; aggregate-ICE schema (no petrol/diesel split). Credentials via THAILAND_AIU_THAIAUTO_USER/_PW. The API is port 3000 — GHA runners reach it directly. Whole was rebased onto the AIU passenger+pickup definition (old series archived in data/Thailand_legacy.csv). See docs/architecture/29-source-thailand.md. |
| .github/workflows/fetch-japan.yml | Runs fetch_japan.py daily from the 1st of each month (08:00 UTC) and on manual dispatch — most runs are no-ops until JADA publishes the file for the previous month (typically the first business week). If data/Japan.csv changes, the workflow commits it and triggers render-country.yml with country=Japan. The same run refreshes the top-brands file market/japan_top.json (and its month store market/japan_months.json) from the workbook's maker rows; a change there alone is committed but never triggers a render. |
| scripts/fetch_indonesia.py | Automated data ingestion for Indonesia. Logs into GAIKINDO's ProjectSend portal (files.gaikindo.or.id, csrf + password login, credentials via INDONESIA_GAIKINDO_USER/_PW), discovers the newest cumulative "Wholesales Jan-XXX YYYY" PDF (English and Indonesian month abbreviations) and parses its seven model-level Excel-paste sheets positionally with pdfplumber (1.56 pt font — tight tolerances, split-number reassembly, wrapped-row re-attachment). Whole = GAIKINDO Passenger Car (Sedan+4x2+4x4+LCGC, continues the R. Andrew series) → data/Indonesia.csv; plus Pickups (pick ups < 5 t + double cabins), HDV (trucks ≥ 5 t), Buses. Fuel split G/D/BEV/HEV/PHEV per model; rarities (CNG) → OTHERS. Every month is checksummed against the PDF's printed section totals and PC/CV/DOMESTIC summary rows — any mismatch aborts before writing. Each cumulative file rewrites all covered months (absorbs revisions). See docs/architecture/30-source-indonesia.md. |
| .github/workflows/fetch-indonesia.yml | Runs fetch_indonesia.py daily from the 10th of each month (09:35 UTC) and on manual dispatch (download_url override, force) — the script self-throttles via the newest portal file title, so runs are no-ops until GAIKINDO publishes a new month. If a CSV changes, the workflow commits the four Indonesia CSVs and triggers render-country.yml with country=Indonesia (variant Whole). |
| scripts/build_theme.py | Extracts the design tokens (palette + type stack, light and dark) from index.html's :root blocks into assets/theme.css, which the generated standalone pages — sources/*.html and schedule*.html — link. index.html stays the single source of truth for the design, so those surfaces cannot drift away from the gallery the way they did before #221. --check verifies the committed stylesheet still matches and fails CI if not. assets/theme.css is generated — never hand-edit it. |
| scripts/snapshot_builder.py | Snapshots the Builder-tab aggregated BEV/ICE/PHEV curves (world + 13 regional/weight-based groups) to builder_history/<date>.csv plus a metadata entry in builder_history/index.json. Mirrors the in-page Builder math (including the v1=0 anchor recovery), so a snapshot reproduces what the page plots that day. Every snapshot is on one x-basis: the formerly one-year-late band was rebuilt from git rather than annotated; see #219. Regression tests in scripts/test_snapshot_builder.py. |
| scripts/rebuild_builder_history.py | Regenerates any past snapshot from the params.csv / weights.csv git holds for that date. --cohort additionally writes builder_history/cohort/, every frame restricted to the 44 countries present on all dates — which separates "the model changed its mind" from "the gallery gained countries". |
| R/build_backtest.R | The backtest: re-fits every country series from data/ truncated to each month, writing backtest/params/<YYYY-MM>.csv + backtest/weights/<YYYY-MM>.csv. Answers "what would this model have said at date X" — a different question from builder_history/, and one that reaches 2015 where git-based archaeology cannot start before 2025-09. Truncates today's revised CSVs, so it is an upper bound, not a clean out-of-sample test. Resumable and parallel; the 2015 backfill is ~9,200 fits (~1 h on 4 cores), each new month ~95. A newly added country is backfilled into every existing month once, automatically. |
| scripts/build_backtest_series.py | Pivots backtest/ into backtest/series/<group>.json — one group, all months, both country sets, at half-year resolution, with the 20/50/80 % crossing years precomputed per frame. This is the only form the browser reads. Imports its aggregation from snapshot_builder.py so a backtest curve and a snapshot curve cannot differ by code. |
| scripts/build_builder_series.py | The same pivot for builder_history/. Nothing renders its output today, but it owns the grid/sampling/labelling helpers the backtest builder imports, so the two aggregations cannot drift. |
| scripts/build_builder_gif.py | Renders each group's series into backtest/series/<group>.gif (Pillow, no matplotlib). Only the curated GIF_GROUPS (four blocs + six spotlight countries), overwritten in place — each run is the same animation one frame longer, so vintages are not kept. Quarterly subsample (--step 3), so 116 monthly frames become a 40-frame, ~12 s loop rather than a minute of footage. Every frame carries the revised-data caveat, because a still travels without the page around it. |
| .github/workflows/snapshot-builder.yml | On the 25th of each month (09:00 UTC) and on manual dispatch: runs snapshot_builder.py to freeze what the page shows today into builder_history/, and extends the backtest by the new month, rebuilds backtest/series/ and re-renders the animations. Both land in one commit, so the Time-lapse panel in the Builder tab is never behind the data. Dispatch inputs: date, backtest_only, and refit (`Country |
| scripts/check_country_integration.py | Verifies every country in data/ is wired into every place the gallery needs it — Submit-Data list, flag file and emoji, continent colour, a Builder region group (and that the two group mirrors agree), time zone, source page, glossary, its fetch workflow's docs and schedule labels, and a backtest history for the Time-lapse. Pre-existing gaps are listed with reasons in KNOWN_GAPS; a fixed gap left there also fails. The checklist it enforces: docs/architecture/42-adding-a-country.md. |
| .github/workflows/backfill-bands.yml | Manual only: recomputes bands/<slug>.json for every series (or the given countries) via scripts/backfill_bands.R and commits only bands/. For the first fill and after changing R/bands.R. |
| .github/workflows/runner-canary.yml | Monthly (20th) and on dispatch: runs the fetchers most exposed to an OS change (OCR, pdftotext, Playwright), one render and the system-Python step on the pinned ubuntu-24.04 image and on the next one, and compares the results via scripts/runner_canary.py. Read-only; red when the next image would change or break something. Risk register: docs/architecture/56-runner-upgrades.md. |
| .github/workflows/preview-render.yml | On every pull request touching R/ (and on manual dispatch): renders a fixed set of series twice from the same data — with the PR's R/ and the base branch's — and uploads the side-by-side charts plus params.csv before/after as the render-preview artifact, linked from one PR comment. Commits nothing. |
| scripts/build_render_preview.py | Turns the preview's per-series renders into compare/*.png (base | PR, side by side), index.html and a summary table. |
| .github/workflows/check-country-integration.yml | Runs check_country_integration.py on every pull request touching data, index.html, R, flags, docs, backtest/params or a fetch workflow (and on manual dispatch). |
| scripts/fetch_uruguay.py | Automated data ingestion for Uruguay. Scrapes the ACAU homepage for the current year's "Compilado YYYY" xlsx (filename is timestamp-based and changes between publications), parses the AUTOS + SUV sheets, maps the per-model Combustible codes (E→BEV, PHEV→PHEV, H→HEV, N→PETROL, D→DIESEL, MHEV→OTHERS), and upserts new months into data/Uruguay.csv. Cross-checks against each sheet's bottom TOTAL row. Self-throttles via the CSV's latest period. |
| .github/workflows/fetch-uruguay.yml | Runs fetch_uruguay.py daily from the 1st of each month (08:10 UTC) and on manual dispatch — most runs are no-ops until ACAU edits the previous month into the Compilado workbook (typically the first week of the following month). If data/Uruguay.csv changes, the workflow commits it and triggers render-country.yml with country=Uruguay. The same run refreshes market/uruguay_top.json (+ market/uruguay_months.json) from the workbook's per-model rows; a change there alone never triggers a render. |
| scripts/fetch_georgia.py | Automated data ingestion for Georgia (quarterly). Reads Geostat's «Automobile» statistics portal through its public JSON API (autoapi.geostat.ge, no key): the period picker for the published quarters, the quarter's fuel split of initial registrations (every vehicle category, new and imported used — the scope the hand-entered series always had) and the treemap of brands and models. Electric → BEV, Hybrid → HEV (one combined bucket), Gasoline + Gasoline-gas → PETROL, Diesel → DIESEL, Gas + Others → OTHERS; rows on the quarter's middle month. Checks Geostat's year figure against the sum of its quarters and the new quarter against a year earlier; appends new quarters only. Writes data/Georgia.csv and market/georgia_top.json. Tests: scripts/test_fetch_georgia.py. See docs/architecture/52-source-georgia.md. |
| .github/workflows/fetch-georgia.yml | Runs the fetcher tests, then fetch_georgia.py, daily at 06:55 UTC on the 10th–31st of the two months after each quarter (Jan/Feb, Apr/May, Jul/Aug, Oct/Nov) and on manual dispatch (quarter, backfill, force). Self-throttles with one request (the period picker) when the newest quarter is already in the CSV. Commits data/Georgia.csv + market/georgia_top.json and dispatches render-country.yml once. |
| scripts/fetch_south_africa.py | Automated data ingestion for South Africa (quarterly). Lists naamsa's «Quarterly Review of Business Conditions» PDFs through the naamsa.net WordPress media API, reads the NEV drivetrain table and the section-7 aggregate passenger + commercial sales with pdftotext -layout. Electric → BEV, Plug-in hybrid → PHEV, Traditional hybrid → HEV, TOTAL = passenger + commercial, ICE = TOTAL − NEVs; rows on the quarter's middle month. Scope is the whole new-vehicle market (cars, bakkies/minibuses, trucks, buses) — not M1. Checks column sums, reconciles quarters with the annual column (undoing a PHEV↔HEV row swap), guards completeness. Writes data/South Africa.csv. Tests: scripts/test_fetch_south_africa.py. See docs/architecture/60-source-south-africa.md. |
| .github/workflows/fetch-south-africa.yml | Runs the fetcher tests, then fetch_south_africa.py, daily at 12:25 UTC on the 10th–31st of Feb, May, Aug and Nov and on manual dispatch (backfill, force). Self-throttles with one JSON request when the newest review's quarter is already in the CSV. Commits data/South Africa.csv and dispatches render-country.yml once. |
| scripts/fetch_vietnam.py | Automated data ingestion for Vietnam — new passenger-car sales (≈ M1), not registrations. Reads VAMA's monthly cover letter (pdftotext -layout: total industry and its passenger / commercial / special-purpose split, the members' BEV and Hybrid subtotals) and Detail sheet (pdfplumber: members' models, for the mild hybrids and the brand/model tables), VinFast's monthly delivery release on vingroup.net and TC Group's Hyundai table; stores every figure with its URL in data/raw/vietnam/components.csv and writes data/Vietnam.csv once all three are in (BEV = VinFast + VAMA BEV, HEV / MHEV per model, non-member imports = OTHERS, Hyundai = ICE). Revises VinFast's previous month from the year to date; rebuilds months whose Detail sheet VAMA mislabelled. Also writes market/vietnam_top.json / vietnam_all_top.json + month stores. Tests: scripts/test_fetch_vietnam.py. See docs/architecture/64-source-vietnam.md. |
| .github/workflows/fetch-vietnam.yml | Runs the fetcher tests, then fetch_vietnam.py, twice daily (04:05 & 10:05 UTC) on the 8th–28th, on manual dispatch (period, direct URLs, rebuild, force) and on a push of checks/vietnam_reference.csv; then scripts/check_vietnam_reference.py → market/vietnam_checks.json. Commits the CSV, the components file and the market files and dispatches render-country.yml for Whole when the CSV changed. |
| checks/vietnam_reference.csv + .github/workflows/vietnam-reference-reminder.yml | Hand-kept reference figures for Vietnam (monthly media totals, the police's half-year and year registrations) that the fetcher compares with; the reminder workflow (daily 02:45 UTC, 12th–28th and 1st–11th of January/July) keeps one issue labelled vietnam-reference open while a figure is due and closes it when none is. |
| scripts/fetch_germany.py | Automated data ingestion for Germany (registrations). Discovers the newest KBA monthly press-release …_merkmale.xlsx (listing page → …_komplett.html → SharedDocs xlsx, both hops overridable), reads the "Kraftstoffarten" block (Benzin→PETROL, Diesel→DIESEL, Elektro→BEV, Hybrid − darunter Plug-in → HEV, Plug-in→PHEV), and upserts one monthly row into data/Germany.csv (Whole). OTHERS is the residual TOTAL − BEV − PHEV − HEV − PETROL − DIESEL (LPG/CNG/H₂/fuel-cell/Sonstige) — matches the pre-automation hand-entry convention exactly (verified against 08/2026). Line-level upsert keyed on (period, Whole); historical quarterly/modelled decimal rows are preserved byte-for-byte. Honours KBA_FETCH_RELAY/KBA_PROXY for datacenter-IP blocks; --file/--url/--dry-run/--force. See the module docstring and docs/architecture/country_source_stubs.yaml (Germany). |
| .github/workflows/fetch-germany.yml | Runs fetch_germany.py daily on the 4th–9th of each month (06:00 UTC) and on manual dispatch (url, force, dry_run). The fetcher early-exits once the previous month is in the CSV, so most runs are no-ops until KBA publishes the press release (typically the first week). If data/Germany.csv changes, the workflow commits it and triggers render-country.yml with country=Germany (variant Whole) — unlike the Germany fleet fetcher, which does not render. |
| scripts/fetch_turkey.py | Automated data ingestion for Türkiye. Finds the TÜİK "Motorlu Kara Taşıtları" bulletin for the target month itself — the Veri Portalı is a React SPA with no listing endpoint, so scripts/tuik_discover.py walks bulletin ids outward from the newest one in the CSV and matches on title + period — then pulls the authoritative monthly + YTD totals from the narrative, OCRs the fuel-breakdown table image (poppler + imagemagick + tesseract-tur), and upserts the row into data/Türkiye.csv (Elektrik→BEV, Hibrit→HEV, Benzin→PETROL, Dizel→DIESEL, LPG→OTHERS). Three validation layers: narrative-vs-OCR Toplam, per-fuel sum check with single-error auto-repair via Pay % cross-check, and previous-year-same-month column cross-check against the existing CSV row. Self-throttles via the CSV's latest period. |
| .github/workflows/fetch-turkey.yml | Runs fetch_turkey.py daily from the 15th of each month (08:30 UTC) and on manual dispatch. No input is required — the bulletin id is auto-discovered; press_id / pdf_url remain as overrides for when the portal changes shape. Runs self-throttle to a no-op until TÜİK publishes the next bulletin (date footnoted in the previous one as "Bu konu ile ilgili bir sonraki haber bülteninin yayımlanma tarihi …"). If data/Türkiye.csv changes, the workflow commits it and triggers render-country.yml with country=Türkiye. |
| scripts/fetch_uk.py | Automated data ingestion for the UK. Finds SMMT's monthly new-car release (compiled from DVLA data) through the site's WordPress REST API — the post of the publication window whose fuel table is for the month; SMMT's vehicle-data page is the fallback — and reads its HTML table by row label: BEV / PHEV / HEV / PETROL (incl. mild hybrids) / DIESEL / TOTAL → data/UK.csv (Whole). Checks the sums, recomputes the printed shares and % changes (a shifted column fails), compares year-to-date minus the CSV's earlier months with the month and the total with the same month a year earlier. Appends new months only; the hand-entered history (2015-01 → 2026-09, same source SMMT) is never rewritten. --from-html parses a saved page offline. Writes market/uk_top.json + uk_months.json (marque table and top-10 models, every powertrain). Regression tests: scripts/test_fetch_uk.py. See docs/architecture/50-source-uk.md. |
| .github/workflows/fetch-uk.yml | Runs the fetcher tests, then fetch_uk.py, twice daily on the 1st–15th (10:25 & 16:25 UTC — SMMT releases at 09:00 UK time on about the 4th working day) and on manual dispatch (period, backfill, force). Self-throttles without any HTTP request once the month is in every variant's CSV and the top file is current; a release still missing on the 11th fails the run. Commits data/UK.csv + market/uk_top.json + market/uk_months.json, then dispatches render-country.yml once. |
| scripts/fetch_norway.py | Automated data ingestion for Norway. Finds OFV's monthly new-car release (compiled from the Statens vegvesen register) through OFV's RSS feed (sitemap fallback) — the article whose "Fordeling per drivstoff" table is for the month, the van release of the next day skipped — maps its seven fuel rows to BEV / PHEV / HEV / PETROL / DIESEL / OTHERS, and checks the month's total and the year-to-date fuel mix exactly against OFV's open api/statistikk/export.json (CC BY 4.0). With no article by the 10th the month is derived from that export (flagged in notes). Writes data/Norway.csv line-level (replacing ACEA's later copy of the same month) and market/norway_top.json (+ norway_months.json) from OFV's top-30 brand and model tables. See docs/architecture/54-source-norway.md. |
| .github/workflows/fetch-norway.yml | Runs the fetcher tests, then fetch_norway.py, twice daily on the 1st–12th (07:10 & 15:10 UTC — OFV releases early on the first working day) and on manual dispatch (period, force). Commits data/Norway.csv and the two market/norway_* files, then dispatches render-country.yml once. |
| scripts/fetch_canada.py | Automated data ingestion for Canada. Reads Statistics Canada's Web Data Service (WDS cube 20-10-0025 "New motor vehicle registrations"): one getCubeMetadata call resolves the dimension members by name (Geography=Canada, Vehicle type, the Fuel type leaves), then a batched getDataFromCubePidCoordAndLatestNPeriods call pulls the latest quarters per fuel. Maps the fuel leaves (battery electric→BEV, plug-in hybrid→PHEV, hybrid→HEV, gasoline→PETROL, diesel→DIESEL, other/fuel-cell→OTHERS), stores each quarter under its middle month (Q1→02 … Q4→11). Produces three variants from this light-vehicle cube: Whole = EU M1 (Passenger cars + Multi-purpose vehicles/SUVs, harmonised with how every other country counts passenger cars) → data/Canada.csv; plus Canada-specific Pickups (Pickup trucks) → data/Canada_Pickups.csv and Vans (minivans + cargo vans) → data/Canada_Vans.csv. Definition change: Whole was historically passenger-cars-only; it is now M1 (starts ~2017 where the SUV fuel split begins). See docs/architecture/17-source-canada.md. Full logic in the module docstring and docs/architecture/17-source-canada.md. |
| .github/workflows/fetch-canada.yml | Runs fetch_canada.py daily on the 8th–20th of March/June/September/December (06:40 UTC) — the cube is quarterly and StatCan releases it ~2.5 months after quarter-end — and on manual dispatch. The script always re-fetches the latest N quarters (StatCan revises recent ones); the commit step is change-gated, so steady-state runs are a no-op. Detects which per-variant CSVs changed and triggers render-country.yml once with the touched variants (country=Canada), which it renders serially in a single "Render: Canada" run. |
| scripts/fetch_nepal.py | Automated data ingestion for Nepal. Downloads the Department of Customs' cumulative monthly Foreign Trade Statistics workbooks (customs.gov.np — three-hop discovery: homepage nav → fiscal-year category → content page → one xlsx per Nepali month), parses sheet "5_Imports_By_Commodity" (8-digit HS codes) and rebuilds every month of the processed fiscal year from consecutive-file deltas. Whole = HS 8703 cars/jeeps/vans (≈ M1, imports not registrations — Nepal imports its entire market) → data/Nepal.csv; passenger three-wheelers (petrol auto-rickshaws + electric e-rickshaws, 16.5k units FY 2081/82!) are split into data/Nepal_3-Wheelers.csv. Hybrids use the single-Hybrid-bucket convention (DoC's PHEV/HEV tariff descriptions are displacement-garbled). Nepali fiscal months are labelled by their Gregorian end month (Shrawan → August). Coverage is parsed from each workbook's own English header line — the Devanagari link labels are too inconsistent to trust. See docs/architecture/32-source-nepal.md. |
| .github/workflows/fetch-nepal.yml | Runs fetch_nepal.py daily (07:50 UTC) and on manual dispatch (fy, backfill inputs). The fetcher cheap-skips when the CSV already carries every workbook published for the current fiscal year, so steady-state runs are no-ops; when a CSV changes, the workflow commits it and triggers render-country.yml once with both variants (Whole, 3-Wheelers), which it renders serially in a single "Render: Nepal" run. |
| scripts/fetch_usa.py | Automated data ingestion for the USA. Auto-discovers the latest ANL "Total Sales for Website" PDF on the ESIA reference page (or takes a direct --pdf-url), parses the full monthly Light-Duty table (BEV/PHEV/HEV/Total LDV; ICE = Total − BEV − PHEV − HEV, OTHERS = 0), and upserts a trailing window of --months (default 3) months into data/USA.csv. ANL revises the last ~2 months between releases, so the window appends new months and corrects recently-revised ones in place; rows older than the window are never touched (deep-history rows may still differ from a newer PDF). The CSV is only rewritten when a value in the window actually changed, so steady-state runs are a no-op. |
| scripts/fetch_argentina.py | Automated data ingestion for Argentina. Reads DNRPA's free record-level Inscripciones iniciales de autos open data (datos.jus.gob.ar CKAN — one record per first registration, 2018 →), keeps 0 km registrations, and writes Whole (passenger-car body types ≈ M1), Private/Industry (natural vs legal-person first owner) and Pickups from one download — Pickups is data only (CSV kept current, never rendered). The records carry no fuel field: the powertrain is classified from the model designation by the hand-edited, tested rule table classification/argentina_rules.csv (e.g. Arkana "E-Tech Hybrid" and Suzuki "Hybrid" are mild hybrids; Volvo T8 / BMW 330E / BYD DM-i are plug-ins), validated against ACARA's electrified totals. Every run also regenerates the full designation → class → rule mapping and the top BEV/PHEV brands and models (classification/argentina_models.csv, argentina_top.json), both shown on the Argentina source page. See docs/architecture/39-source-argentina.md. |
| scripts/market_top.py | Shared builder for the per-country top brands / models summary (market/<slug>_top.json; Argentina's classification/argentina_top.json): trailing-12-month units per electrified class and, where the CSV splits them, petrol and diesel; top 10 brands and top 15 models, country-neutral schema, byte-stable. Used by the Argentina, Spain (DGT), Malaysia (data.gov.my) and Ukraine (MIA register) fetchers; rendered on each source page as leaderboard grids ("Who sells the cars" — one column per powertrain, Share / Units toggle) via the market_breakdown: front-matter key. Refresh failures are warnings and never block a data commit. Tests: scripts/test_market_top.py. See docs/architecture/03-data-objects.md §3.16. |
| .github/workflows/fetch-argentina.yml | Runs the classifier tests, then fetch_argentina.py, twice daily on the 8th–25th (09:15 & 21:15 UTC; DNRPA uploads ~the 11th) and on manual dispatch (variant, period, backfill, force). Self-throttles once every CSV has the month; real runs re-derive the current and previous year (DNRPA re-uploads past years) and dispatch render-country.yml once with the touched rendered variants (Whole/Private/Industry; Pickups is data only). |
| scripts/fetch_ukraine.py | Automated data ingestion for Ukraine. Reads the Ministry of Internal Affairs' free record-level vehicle register on data.gov.ua (CKAN — one record per registration operation, yearly zips, the current year re-uploaded around the 1st), keeps first registrations by the register's own operation codes and writes Whole (new passenger cars, M1), Private/Industry (owner type P/J), Used (used imports at first Ukrainian registration) and Vans (new goods vehicles ≤ 3.5 t, N1) from one download, 2018-09 onward. The fuel comes from the record itself (ЕЛЕКТРО → BEV; the register's single "electric or petrol/diesel" value → one combined Hybrid bucket, no PHEV split). Governance checks abort on schema drift, broken uploads, unknown fuel strings or a Private+Industry≠Whole mismatch, and list any unmapped first-registration code in the step summary. Also writes market/ukraine_top.json. Tests: scripts/test_fetch_ukraine.py. See docs/architecture/40-source-ukraine.md. |
| .github/workflows/fetch-ukraine.yml | Runs the fetcher tests, then fetch_ukraine.py, twice daily on the 1st–15th (08:40 & 20:40 UTC) and on manual dispatch (variant, period, backfill, force). Self-throttles once every CSV has the month; a real run reads the current and previous yearly file and dispatches render-country.yml once with the touched variants. |
| scripts/fetch_hong_kong.py | Automated data ingestion for Hong Kong. Reads the Transport Department's free record-level Particulars of first registered vehicles on DATA.GOV.HK (CKAN — one CSV per month from 2019-11, one row per vehicle: class, make, model, fuel, TD first-registration status, gross weight), keeps private cars never registered abroad (status A/B/C1 → Whole), used imports (C2 → Used) and light goods vehicles ≤ 3.5 t (Vans). BEV from TD's fuel field; plug-in hybrids from the model designation (PHEV_RULES — the fuel field has no hybrid value, full/mild hybrids stay in PETROL). Cross-checks every run against TD's aggregate table 4.1(e) and aborts on a real mismatch; writes market/hong_kong_top.json. Line-level upserts into data/Hong Kong.csv, _Used, _Vans. See docs/architecture/41-source-hong-kong.md. |
| .github/workflows/fetch-hong-kong.yml | Runs the fetcher tests, then fetch_hong_kong.py, twice daily (03:20 & 11:20 UTC — TD uploads month M between the 13th and 28th of M+1) and on manual dispatch (variant, period, backfill, force). Self-throttles on the portal's newest month (one JSON request, no download); a real run reads the newest 12 monthly files. Commits changed CSVs + the top list and dispatches render-country.yml once with the touched variants. |
| scripts/fetch_peru.py | Automated data ingestion for Peru. Reads SUNARP first registrations as compiled by the Asociación Automotriz del Perú from AAP's public BI-AAP Power BI report, through the report's anonymous query API (resource key from the embed URL on aap.org.pe/estadisticas/biaap, API host from the embed page — no login): one query per month returns brand × model × class × body × powertrain from 2019-01. Cars, station wagons, SUVs and MPVs → Whole (M1); pickups, chassis-cabs, box and panel vans → Vans (N1); BEV / PHEV / HEV (full + mild) / petrol / diesel / CNG / LPG from AAP's powertrain class. Writes a month only when it is before the report's reload month, classified by AAP and confirmed by AAP's printed monthly report (the newest month of each reload is a preliminary mid-month cut); re-reads the whole history on every real run because AAP re-classifies the powertrain split of earlier months, and lists the revised rows in the step summary; self-checks every month against an independent aggregate query, cross-checks light + heavy totals against the printed report (exact for all 90 months), line-level upserts into data/Peru.csv / data/Peru_Vans.csv, and writes market/peru_top.json (top BEV / PHEV / HEV / MHEV brands and models). Offline mode --from-json (rows saved with --dump-json) for tests and rebuilds. Regression tests: scripts/test_fetch_peru.py. See docs/architecture/45-source-peru.md. |
| .github/workflows/fetch-peru.yml | Runs the fetcher tests, then fetch_peru.py, daily at 15:45 UTC (AAP refreshes the report on no fixed day) and on manual dispatch (variant, period, backfill, force). Self-throttles on the newest classified month and the report's reload stamp (kept in market/peru_top.json; a few small requests, no month queries); a real run — after each AAP reload, or when a held month's printed report appears — re-reads the whole history (~90 queries, ~1 min). Commits changed CSVs and the top file, then dispatches render-country.yml once with the changed variants. |
| scripts/fetch_new_zealand.py | Automated data ingestion for New Zealand. Asks NZTA Waka Kotahi's open Motor Vehicle Register (an ArcGIS feature service, URL resolved from its Hub item) for grouped counts — first registrations by month, import status, vehicle class and motive power — and maps them to BEV / PHEV (EREV folded in) / HEV / PETROL / DIESEL / OTHERS → data/New Zealand.csv (Whole = new M1), New Zealand_Vans.csv (new N1), New Zealand_HDV.csv (new N2+N3), and New Zealand_Used.csv (used-import M1). The series as it was until 2026-10 (new + used light vehicles, which the register reproduces to ~1 %) is parked unchanged in data/New Zealand_legacy.csv — an archive, not fetched or rendered. Also writes market/new_zealand_top.json (Whole) and market/new_zealand_used_top.json (Used). Offline replay with --from-json. See docs/architecture/19-source-new-zealand.md. |
| .github/workflows/fetch-new-zealand.yml | Runs the fetcher tests, then fetch_new_zealand.py, twice daily on the 3rd–20th (04:55 & 18:55 UTC — NZTA reloads the register around the 5th) and on manual dispatch (variant, period, since, force, dry_run). Self-throttles without any HTTP request once the month is in the CSV and the top file is current; a register still not reloaded on the 20th fails the run. Dispatches render-country.yml once with the changed variants. |
| scripts/fetch_paraguay.py | Automated data ingestion for Paraguay. Reads the customs administration's (DNIT, ex Aduanas) free record-level open data on datosabiertos.aduana.gov.py — one item-level CSV per month (≈ 300 MB, every import and export declaration item; fetched as 12 parallel byte ranges because the server is slow per connection). Paraguay builds no cars, so imports are the market: keeps NCM 87.03 cars imported for consumption (bonded-warehouse and free-zone entries are not counted, the exit is — nothing twice) and writes Whole (declared NUEVO, rendered) and Used (USADO, data only — never rendered) from 2017-01. The powertrain is the HS 2017 tariff subheading (8703.80 BEV, .60/.70 PHEV, .40/.50 HEV incl. mild, .2x petrol, .3x diesel). One item = one car (the quantity column is unreliable; believed only with the weight of that many cars). Also writes market/paraguay_top.json + month store. Line-level upserts; offline mode --from-dir. See docs/architecture/47-source-paraguay.md. |
| .github/workflows/fetch-paraguay.yml | Runs the fetcher tests, then fetch_paraguay.py, twice daily on the 1st–7th (16:55 & 22:55 UTC — the portal regenerates month M's files on the 1st of M+1) and on manual dispatch (variant, period, backfill, force). Self-throttles on the newest complete month (file list + one HEAD, no download); a real run downloads the newest two months (~1 min each). Commits changed CSVs + the top list and month store, then dispatches render-country.yml once with the touched variants. |
| scripts/fetch_ecuador.py | Automated data ingestion for Ecuador. Reads the tax authority's (SRI) free record-level open data «Vehículos Nuevos» on descargas.sri.gob.ec — one CSV per calendar year with every processing of an invoiced new vehicle (brand, catalogue description, class, fuel code), the register AEADE's official sales statistics are built from. Counts each vehicle once, in the month of its first processing (two-year lookback, so December sales processed again in January are not counted twice); Whole = classes AUTOMOVIL + JEEP, Vans = CAMIONETA; powertrain from the end of SRI's catalogue description (… EV → BEV, … HYBRID → hybrid, … DIESEL, Nissan e-POWER → hybrid), else the invoice's fuel code; hybrids are split into PHEV / EREV / HEV by the hand-edited rule table classification/ecuador_rules.csv (every hybrid designation and its rule in the generated classification/ecuador_models.csv; checked against ZEMO's and AEADE's plug-in counts). Cross-checks every real run against AEADE's monthly press bulletin (total, BEV, car + SUV; abort beyond 10 %). Also writes market/ecuador_top.json. Offline mode --from-dir / --bulletin. See 48-source-ecuador.md. |
| .github/workflows/fetch-ecuador.yml | Runs the fetcher tests, then fetch_ecuador.py, twice daily on the 1st–14th (14:05 & 23:05 UTC — SRI regenerates the current year's file early in the month) and on manual dispatch (variant, period, backfill, force). Self-throttles on the newest complete month (one HEAD, no download); a real run downloads the newest two yearly files plus the two before them (≈ 300 MB) and re-derives both years. Commits data/Ecuador*.csv + market/ecuador_top.json, then dispatches render-country.yml once with the changed variants. |
| scripts/fetch_switzerland.py | Automated data ingestion for Switzerland (with Liechtenstein). Reads ASTRA's free record-level extracts of the national vehicle register IVZ (opendata.astra.admin.ch/ivzod — NEUZU.txt, every new registration of the running year, refreshed on the 1st and mid-month; GEBR.txt, used imports; closed years as NEUZU-<YYYY>.txt / GEBR-<YYYY>.txt), one tab-separated row per vehicle with its EU class, vehicle kind, fuel and hybrid code. Personenwagen (M1) → Whole; N1 Lieferwagen → Vans; N2/N3 lorries + tractor units → HDV; M2/M3 over 3.5 t → Buses; L1 + L3 → 2-Wheelers; used-import cars → Used. PHEV/HEV from the hybrid code (CO2/electric consumption where a national type approval has none); every filter calibrated against ACEA (Whole 2025-01..2026-08: −0.02 %). A live month is written once, ACEA-style: late registrations of the year's written months flow into the next month, nothing is revised; closed years are counted by registration month (--backfill). Reads only the snapshot's first 16 KB when there is nothing to do. --acea-check compares the newest months with ACEA's release. Line-level upserts; writes market/switzerland_top.json. Regression tests: scripts/test_fetch_switzerland.py. See docs/architecture/46-source-switzerland.md. |
| .github/workflows/fetch-switzerland.yml | Runs the fetcher tests, then fetch_switzerland.py, twice daily on the 1st–6th (09:20 & 15:20 UTC — ASTRA posts the new snapshot on the 1st) and on manual dispatch (variant, backfill, backfill_from, force). Commits changed CSVs and the top files, then dispatches render-country.yml once with the changed variants. |
| .github/workflows/check-switzerland-acea.yml | Daily on the 23rd–28th (ACEA's release window) and on dispatch: fetch_switzerland.py --acea-check compares the newest ASTRA months of data/Switzerland.csv with ACEA's monthly release and annotates any gap beyond 1 % (total) / 3 % (fuel) as a warning. Writes nothing. |
| scripts/fetch_taiwan.py | Automated data ingestion for Taiwan. Reads the Highway Bureau's (THB) monthly new registrations by vehicle kind × fuel as republished in the MOTC statistics database (statis.motc.gov.tw, table Seq 104, through the JSON endpoint the database's own page calls — THB's own statistics site blocks datacenter IPs). 小客車 (passenger cars, M1) → Whole; 小貨車 (goods ≤ 3.5 t) → Vans; 大貨車 (goods > 3.5 t) → HDV; 大客車 (buses and coaches) → Buses. BEV / PHEV / EREV / HEV / petrol / diesel / LPG straight from THB's thirteen fuel categories, from 2021-01 (the first month with a consistent plug-in coding). Ids are resolved from the page's labels and re-checked against the JSON; fuels must sum to THB's total; every month is cross-checked against THB's separately compiled brand table (Seq 118). Re-reads the newest 24 months (revisions listed); --dump-dir / --from-dir for offline replays. Line-level upserts; writes market/taiwan_top.json (brands of new passenger cars, every powertrain). Regression tests: scripts/test_fetch_taiwan.py. See docs/architecture/49-source-taiwan.md. |
| .github/workflows/fetch-taiwan.yml | Runs the fetcher tests, then fetch_taiwan.py, twice daily on the 12th–28th (01:45 & 09:45 UTC — MOTC publishes a month around the 15th of the next) and on manual dispatch (variant, period, backfill, force, debug_dump — the last uploads every MOTC response as an artifact for offline debugging). Self-throttles on the newest month in the table page (one request). Commits data/Taiwan*.csv + market/taiwan_top.json, then dispatches render-country.yml once with the changed variants. |
| scripts/fetch_colombia.py | Automated data ingestion for Colombia. Scrapes ANDI's Cámara Automotriz page for the newest joint ANDI/FENALCO "Informe del Sector Automotor" PDF (RUNT-sourced; the filename shape has changed every year — discovery reads month + year out of any bulletin-named PDF link rather than pinning a template). Since September 2026 that page lists only year-end reports, so when the newest listed bulletin is older than the month due, discovery falls back to rebuilding that month's andi.com.co/Uploads/ URL, where ANDI still serves it — accepted only if the body really is a PDF and parses to the month requested. If neither route reaches the due month, the run exits quietly before the 20th and fails after it, so a source that moves can never look green. Then runs pdftotext -layout and reads the three monthly bar charts (passenger-car total, eléctricos → BEV, híbridos → HEV as a single combined hybrid bucket; ICE = TOTAL − BEV − HEV) for the ~31 months each PDF covers. Bar values that pdftotext puts below their month label are picked up by a column-aware look-ahead; a value that still cannot be read is unknown — it never overwrites an existing CSV cell and never becomes a 0. Regression tests in scripts/test_fetch_colombia.py. See docs/architecture/18-source-colombia.md. |
| .github/workflows/fetch-colombia.yml | Runs fetch_colombia.py daily on the 5th–25th of each month (07:30 UTC) and on manual dispatch (pdf_url override, force, dry_run, dump_listing). dry_run=true parses the newest bulletin, prints what would change plus the full pdftotext output into the log, and uploads PDF + text as a run artifact without writing or committing — the way to check discovery or a parser change from a branch. dump_listing=true additionally reports what the Cámara page actually contains (every PDF href, every AUTOMOTOR mention, and the links it rejected) and saves the page HTML into the artifact — the way to tell “the bulletin moved” apart from “we failed to recognise it”. If data/Colombia.csv changes, the workflow commits it and triggers render-country.yml with country=Colombia. |
| scripts/fetch_czechia.py | Automated data ingestion for Czechia. Streams the Ministry of Transport's free open-data extract of the national vehicle register RSV (download.dataovozidlech.cz/vypiszregistru/vypisvozidel — one row per vehicle ever registered, ~19 M rows, ~7.5 GB with gzip, regenerated on the 1st) without storing it, after a header-only probe of the extract date. New vs used from the register's two first-registration dates; variants by EU category: Whole (M1/M1G), Used, Vans (N1), HDV (N2/N3), Buses (M2/M3). Plug-ins from the hybrid class, CO2 ≤ 60 g/km or the type-approval variant. Whole rows carry exact BEV/PHEV/OTHERS/TOTAL and an estimated HEV/PETROL/DIESEL split until ACEA's always-list write replaces them. Also writes market/czechia_top.json / czechia_used_top.json. Tests in scripts/test_fetch_czechia.py. See docs/architecture/51-source-czechia.md. |
| scripts/fetch_greece.py | Automated data ingestion for Greece (Whole). Finds SEAA's monthly files (the importers' association, which processes ELSTAT's registration data and is ACEA's source for Greece) through the WordPress media API (seaa.gr/wp-json/wp/v2/media): TOTAL from <Y>-<M>-comp.xlsx, BEV/PHEV from the BEV.pdf/PHEV.pdf summary lines (electric.pdf before 2024), HEV/PETROL/DIESEL/OTHERS derived from the press release's one-decimal fuel shares (largest-remainder rounding; OTHERS = LPG + CNG) until ACEA publishes the month, whose counted HEV/PETROL/DIESEL then replace them (scripts/acea_split.py, called by both fetchers; source SEAA / ACEA). A month whose statistics files are not out yet is written provisionally from the press release's preliminary counts. Replaces ACEA rows from 2022-01; ACEA keeps Greece only on its conditional list. Also writes market/greece_top.json. Tests in scripts/test_fetch_greece.py. See docs/architecture/53-source-greece.md. |
| .github/workflows/fetch-czechia.yml | Runs the fetcher tests, then fetch_czechia.py, twice daily on the 1st–8th (05:25 & 13:25 UTC) and on manual dispatch (variant, backfill, force). One real run a month (1–2 h stream); concurrency: fetch-czechia. If a CSV changes, commits the Czechia CSVs + market files and dispatches render-country.yml once with the changed variants. |
| scripts/fetch_lithuania.py | Automated data ingestion for Lithuania. Reads Regitra's register statistics (regitra.lt — table 5, every M1/M1G first registered in Lithuania by Regitra's new/used status × the register's fuel field, one XLSX per year from 2024), cross-checked every run against Regitra's separate monthly totals (exact match required). Table 5 has one hybrid number: the PHEV/HEV split is ACEA's counted split for new cars, else the share in Regitra's open register snapshot (EU hybrid category OVC/NOVC; also feeds market/lithuania_top.json / lithuania_used_top.json), else a provisional trailing share written as Regitra (provisional) and upgraded later. Variants Whole (new) and Used (used imports). Tests in scripts/test_fetch_lithuania.py. See docs/architecture/55-source-lithuania.md. |
| .github/workflows/fetch-lithuania.yml | Runs the fetcher tests, then fetch_lithuania.py, twice daily (06:35 & 12:35 UTC) and on manual dispatch (variant, period, backfill, fleet, force). A run with the month final and no provisional row makes no request; concurrency: fetch-lithuania. If a CSV changes, commits the Lithuania CSVs + market files and dispatches render-country.yml once with the changed variants. |
| scripts/fetch_belgium.py | Automated data ingestion for Belgium. Reads the FPS Mobility's (DIV) monthly registration CSVs (datafiles.mobilit.belgium.be/OPENDATA/DGTRSR/MonthlyRegistrations/, CC0; one file per month from 2025-01 plus a 2013–2024 history file) — an aggregate by new/used vehicle, plate, keeper type, use group and fuel. Variants Whole (new cars), Private / Industry (keeper not VAT-registered / VAT-registered), Vans, HDV, Buses, monthly from 2013-01; BEV, PHEV and HEV (full + mild) counted. Tests in scripts/test_fetch_belgium.py. See docs/architecture/59-source-belgium.md. |
| .github/workflows/fetch-belgium.yml | Runs the fetcher tests, then fetch_belgium.py, daily on the 2nd–20th (17:10 UTC) and on manual dispatch (variant, period, backfill, force). A run with the month already in every CSV makes no request; otherwise the monthly files are probed forward by name until a 404; concurrency: fetch-belgium. If a CSV changes, commits the Belgium CSVs and dispatches render-country.yml once with the changed variants. |
| scripts/fetch_estonia.py | Automated data ingestion for Estonia. Reads Transpordiamet's monthly «Infoleht» workbook (transpordiamet.ee — XLS/XLSX, one per month): sheet «Uued sõidukid» lists every new vehicle with EU category, engine type, EU hybrid type (OVC/NOVC-HEV), owner type, make and model; cross-checked every run against the workbook's fuel table and headline totals. Variants Whole (new M1+M1G from 2021-01), Private, Industry, Used (first registrations − new), Vans (N1), HDV (N2+N3), Buses (M2+M3); also writes market/estonia_top.json. Tests in scripts/test_fetch_estonia.py. See docs/architecture/57-source-estonia.md. |
| .github/workflows/fetch-estonia.yml | Runs the fetcher tests, then fetch_estonia.py, twice daily on the 1st–15th (07:05 & 13:05 UTC) and on manual dispatch (variant, period, backfill, force). A run with the month already in every CSV makes no request; concurrency: fetch-estonia. If a CSV changes, commits the Estonia CSVs + market files and dispatches render-country.yml once with the changed variants. |
| scripts/fetch_latvia.py | Automated data ingestion for Latvia. Reads CSDD's first-registration workbooks (csdd.lv — XLSX, full history, replaced monthly): the headline table (cross-check and "is the month out?"), the passenger-car records (make, model, fuel, new/used status, keeper type) and the lorry/bus records (EU category, energy source). Variants Whole (new cars), Private / Industry (keeper FP / JP), Used (imports), Vans (N1), HDV (N2+N3), Buses (M2+M3), monthly from 2015-01; BEV and PHEV counted, HEV not reported (full and mild hybrids are filed under their fuel). Also writes market/latvia_top.json / latvia_used_top.json. Tests in scripts/test_fetch_latvia.py. See docs/architecture/58-source-latvia.md. |
| .github/workflows/fetch-latvia.yml | Runs the fetcher tests, then fetch_latvia.py, twice daily on the 8th–25th (06:20 & 12:20 UTC) and on manual dispatch (variant, period, backfill, force). A run with the month already in every CSV makes no request; until CSDD publishes, only the 50 KB headline file is read; requests go through the fetch relay on Google Cloud Run in europe-north1 (csdd.lv answers only European IPs); concurrency: fetch-latvia. If a CSV changes, commits the Latvia CSVs + market files and dispatches render-country.yml once with the changed variants. |
| .github/workflows/fetch-usa.yml | Runs fetch_usa.py daily from the 10th of each month (10:30 UTC — offset from the 08:00 slot used by other fetchers) and on manual dispatch. If data/USA.csv changes, the workflow commits it and triggers render-country.yml with country=USA. |
| scripts/fetch_israel.py | Automated data ingestion for Israel. Counts monthly new registrations from the Ministry of Transport's open vehicle registry on data.gov.il (CKAN datastore, ~4.15M currently-registered vehicles), scope sug_degem=P (private passenger cars), grouped by the unpadded road-entry month (moed_aliya_lakvish, e.g. "2016-3"). The registry codes regular HEVs as plain petrol — the fetcher joins every row against the official model catalogue (degem-rechev-wltp, technologiat_hanaa_nm: PLUG IN / regular hybrid / electric / conventional) on (make, model code, model year, trim) to recover the HEV and PHEV split; validated against I-VIA monthly reviews. Also has a --probe mode that dumps datastore schema/cross-tabs. See docs/architecture/34-source-israel.md. |
| .github/workflows/fetch-israel.yml | Runs fetch_israel.py daily on the 10th–20th (08:00 UTC) and on manual dispatch (probe/fetch mode, optional YYYY-MM backfill window, commit gate). Self-throttles once the previous month is present; real runs re-count the last 3 months (late registrations trickle into the snapshot). If data/Israel.csv changes, the workflow commits it and triggers render-country.yml with country=Israel. The same run keeps market/israel_top.json (top BEV / PHEV / HEV brands and models, manufacturer translated from Hebrew) current; that file is committed with the data but never triggers a render. |
| scripts/fetch_france.py | Automated data ingestion for France. Resolves the current SDES Immatriculations mensuelles de voitures neuves par motorisation workbook from the StatInfo landing page (the média id rotates each publication), reads the série VP neuves par énergie — the SIV registry statistics behind ACEA/PFA, with a real HEV column and the full énergie split back to 2011 — and upserts data/France.csv (Whole, EU M1). See docs/architecture/36-source-france.md. |
| .github/workflows/fetch-france.yml | Runs fetch_france.py daily on the 18th–end of month (07:40 UTC) and on manual dispatch (url override, diagnose). Change-gated commit; a changed data/France.csv dispatches render-country.yml for Whole. |
| scripts/fetch_south_korea.py | Automated data ingestion for South Korea. Finds the newest «자동차산업 동향» press releases on the MOTIR board, downloads each PDF and parses total domestic sales (표 3) and the eco-friendly-car table (참고 2: hybrid, BEV, plug-in, hydrogen) — three columns per release (year earlier / previous / current, from the header). Writes the current and the revised previous month into data/South Korea.csv (Whole, ICE = total − eco); cross-checks the year-earlier column. See docs/architecture/43-source-south-korea.md. |
| .github/workflows/fetch-south-korea.yml | Runs the parser tests, then fetch_south_korea.py, twice daily on the 12th–25th (03:35 & 09:35 UTC) and on manual dispatch (releases, force, dry_run). Change-gated commit; dispatches render-country.yml for Whole. |