vccs
Composable charting components for Vue 3, ported from Recharts
Sewage-pollution risk forecasts for river and lake swim spots in England, from live storm-overflow data, rainfall forecasts and the river network.
DataPython · JavaScriptGitHub @ethanbuckleyOnlineOwner unconfirmed
Have you tried it? One tap is enough. No sign-in needed.
Owners of public repositories under a personal GitHub account can sign in to verify ownership and manage the introduction themselves. No administrator approval is needed.
Sign in, generate a verification file, and commit it to this repository’s default branch within 30 minutes. Files on other branches or forks do not count. Then check the file below to manage the project without administrator approval. Keep the file to retain your access.
If needed, you will go to sign-in. After signing in, return here and press your chosen verification button again.
Run /nomorevibe verify in your project folder to verify ownership and manage the introduction yourself.
Verify in one minuteAsk about other verification options
Sewage-pollution risk forecasts for river and lake swim spots in England: live overflow status, a verified rainfall-to-spill model, and network transport with die-off and dilution.
Taken from the repository README
Loading the full README. The saved excerpt is shown below.
Probabilistic sewage-pollution risk for river and lake swim spots in England.
The code and the Python package (dipcast) keep the working name dipcast.
Live site: https://swimsignal.co.uk/ (forecasts for 88 named
spots, rebuilt several times a day by a scheduled GitHub Actions job; free to
run, never sleeps). Source: https://github.com/ethanbuckley/swimsignal (MIT).
Every forecast issued is scored later and published on the site's
verification page.Read the full README ↗SwimSignal forecasts the risk of sewage pollution at river and lake swim spots in England, for today and the next four days. It is free, with no adverts and no accounts.
dipcast.It is a forecast, not a water test. A low risk does not mean the water is clean.
For each spot, SwimSignal:
The result is one level for each day: low, moderate, high or very high risk. A scheduled GitHub Actions job rebuilds the site several times a day. Method, below, has the detail.
Outside England, the site lists the official bathing-water advice for Wales,
Scotland and Ireland. Forecasts there wait for the data (see the plans in
docs/).
docs/PREREGISTRATION-2027.md.The figures and their limits are under Method and Known limits.
Live maps of sewage spills already exist: the National Storm Overflow Hub, The Rivers Trust, WaterWatch, Surfers Against Sewage (SAS) and SewageMap. SewageMap shades the river downstream of live and recent spills in England and Scotland. SAS sends an alert after a discharge and keeps it for 48 hours. These show spills that are happening or have happened.
Forecasts exist for designated bathing waters. The Environment Agency's same-day pollution risk forecast covered no inland bathing water in 2026. Islandswim gives a 24-hour score at 942 UK and Irish bathing waters, and at private spots. Wessex Water estimates bacteria hourly from sensors at three river sites.
SwimSignal differs in three ways:
The other tools were checked on their own pages on 3 Oct 2026.
| Source | What | Licence (all open, no key) |
|---|---|---|
| Water company live feeds (9 companies, ArcGIS, via the National Storm Overflow Hub) | Current status of ~14,200 overflows, latest event start/end, record times (fields under "Live feed fields" below) | CC BY 4.0, per company |
| United Utilities EDM event history 2023-2025 (via Stream) | 743,735 discharge events with start/end; training only | CC BY 4.0 |
| EA "Event Duration Monitoring - Storm Overflow - Start/Stop Detailed Data" | One row per discharge, start and stop, for the 10 companies that operate in England, 2024-2025 (2.95 million events for the nine English companies; scripts/import_ea_events.py). Tested as training data on 17 Oct 2026 and not adopted (docs/NINE-COMPANY-MODEL-TEST-2026-10.md); Welsh Water's files are a Welsh test set | OGL v3 |
| Northumbrian Water EDM spill data 2020-2024 (via Stream) | 471,962 discharge events with start/end (scripts/import_stream_events.py, off CI). Held for testing the spill model, not used in any forecast (docs/EARLIER-SPILL-HISTORY-2026-10.md) | CC BY 4.0 |
| Wessex Water Event Duration Monitoring 2020-2023 and 2024 (Wessex Water's ArcGIS) | 934,361 discharge events 2020-2023 under Wessex's own ids; the 2024 layer is read only to map them to the annual returns' ids. Held for testing, not used in any forecast | CC BY 4.0 |
| South West Water Storm Overflow EDM Monthly (via Stream) | 330,278 discharge events, January 2025 to May 2026 (none in December 2025). Held for testing, not used in any forecast | CC BY 4.0 |
| Thames Water Open Data API, discharge alert stream | Start and stop alerts from April 2022 (almost none before December 2022), joined into 74,781 discharges (scripts/import_thames_alerts.py, off CI). Held for testing, not used in any forecast | Thames Water API Portal terms: share and adapt with attribution (CC BY 4.0's wording, unnamed) |
| Stream ID lookup | Company ids to EA permit ids (id_lookup.parquet) | CC BY 4.0 |
| EA storm overflow annual returns 2021-2025 | Spill counts and hours per overflow per year, WFD waterbody, for all 10 companies | OGL v3 |
| EA bathing water quality (bwq) | Classifications (data/raw/bathing_water_classifications.json) and in-season samples | OGL v3 |
The notices each provider asks for are on the site's terms page ("Data sources
and credits"), and data/spots.json, data/overflows.geojson and
data/verification.json carry them in a credits field (data_credits in
scripts/build_site.py); data/verification_live.csv carries them in comment
lines at its top. The MIT licence covers the code, not the data.
LICENSE-DATA.md says which files are under the ODbL (spots-osm.csv and the
candidate list it came from) and why spots.csv must never take OpenStreetMap rows.
The JSON files under data/anypoint/ also carry the credits; each square's
link list links to them.
Dŵr Cymru Welsh Water's storm overflow map reads a public ArcGIS layer
(Spill_Prod__view). The layer carries no licence, and Dŵr Cymru's site asks
for written permission before its content is reused. SwimSignal emailed Dŵr
Cymru on 1 October 2026 and is waiting for its terms. Until then SwimSignal does
not read the layer: Dŵr Cymru's 128 overflows in England appear with annual
spill history only and no "right now" status. Hafren Dyfrdwy is not in Severn
Trent's live feed, and none of its overflows is in SwimSignal's table. The nine
English companies in the National Storm Overflow Hub are all live.
Off-site tests only, not shown on the site: scripts/check_ecoli_wales.py (Wales
plan, task W10) reads E. coli results from Natural Resources Wales' Water Quality
Archive (NRW_DS124903), under the Open Government Licence v3. The archive is
downloaded by hand into the git-ignored cache; no NRW rows are committed. Credit:
"Contains Natural Resources Wales information © Natural Resources Wales and
Database Right. All rights Reserved. Contains Ordnance Survey Data. Ordnance
Survey Licence number AC0000849444. Crown Copyright and Database Right."
Spill model. One row per overflow per day. Target: any discharge that day.
Features: rainfall that day and the previous two, 3/7/30-day totals, an
antecedent precipitation index, peak 1/3/6-hour intensities, season, and the
overflow's long-term spill count and hours from the EA annual return of the
previous year. Layer 1 is a gradient-boosted classifier with monotone constraints
(more rain can never lower the probability). Layer 2 is a per-overflow
calibration: the Gamma-Poisson posterior mean of observed/expected spill-days,
so overflows with lots of history get their own level and the rest stay near
the pooled prediction. Trained on United Utilities 2023-2024, verified on 2025,
See scripts/train.py; the table below is data/processed/verification_2025.csv.
Held-out 2025, 826,725 overflow-days across 2,241 United Utilities overflows, base rate 7.45% of days with a discharge:
| Model | Brier | Log loss | AUC | Brier skill vs climatology |
|---|---|---|---|---|
| SwimSignal (pooled + site calibration) | 0.0445 | 0.153 | 0.930 | 0.33 |
| pooled only | 0.0452 | 0.155 | 0.927 | 0.33 |
| per-site climatology | 0.0669 | 0.245 | 0.768 | 0.00 |
| naive rule: >10 mm in 48 h | 0.0621 | 0.224 | 0.758 | 0.07 |
Reliability is good below 30% forecast probability and mildly overconfident above 70% (forecast 0.85 verifies at 0.78). The table is the CSV written on 13 Sep 2026; the two comparisons that follow were measured on the first run of 12 Sep, when the full model scored 0.0447. Dropping the negative subsampling does not change this (Brier 0.0449), and recency-weighting the site calibration improves it only slightly (0.0445; the model files use recency 0.25, see docs/MODEL-PROVENANCE-2026-10.md), so the residual is a year effect: 2025 had a 7.45% spill-day rate against 10.1% in 2023-24 for the same overflows, and a model fitted on the earlier years over-predicts it. On wet days (more than 10 mm in 48 h, 22% of overflow-days spill) the Brier score is 0.112 against 0.157 for climatology; on dry days 0.023 against 0.038.
Transport. For each upstream overflow: distance along the network (plus a
lake crossing for lake spots, the shortest way through the lake's water), travel time at a fixed reach
velocity of 0.2 m/s (0.05 m/s across lakes), first-order
die-off with T90 = 72 h, and dilution as the square root of the ratio of upstream
network length at the outfall to that at the spot. The river speed, T90 and the
square root were fitted to the EA E. coli samples on 17 Oct 2026, leaving one year
out at a time (docs/TRANSPORT-FIT-TEST-2026-10.md); until then they were 0.5 m/s,
30 h and the plain ratio, set by hand. They are the setting that best ranks river
sample days within each site, not a measured speed or die-off rate. The product is the probability that a spill
there affects the spot. Risk = 1 - prod(1 - p_i w_i). The site runs
forecast_point(..., gauge=False), so the river level shown beside a forecast
does not change it. With gauge=True, the default, it scales the velocity by
the nearest EA gauge's level index, 0.3 m/s at typical low to 1.0 m/s at typical
high (river_velocity); the API server that used it was removed on 20 Oct 2026.
Two options from Dr James Shucksmith's review (9 Oct 2026) are in the code, off
by default: DIPCAST_VELOCITY_MODE=catchment gives each reach its own speed from
its catchment area (Jobson 1996, eq. 14, at the median flow: about 0.13 m/s at
10 km², 0.29 m/s at 1,000 km²), and DIPCAST_DILUTION_MODE=flow replaces the
length ratio by a fixed spill flow over the spot's estimated median flow. Area
and flows come from the upstream network length through three medians fitted on
NRFA gauging stations (scripts/fit_reach_velocity.py; data from the UK
National River Flow Archive). model/transport.py says how each one works.
DIPCAST_DILUTION_MODE=urban gives each overflow its own spill flow, in
proportion to the ONS built-up area apportioned to it
(scripts/make_overflow_urban_area.py; OGL). It scored worse than the default on
the held-out E. coli tests (docs/URBAN-DILUTION-TEST-2026-10.md).
DIPCAST_LAKE_PATH_MODE (default lake since 11 Oct 2026) treats each WFD lake
between an overflow and the spot as the spot's own lake is treated: its length at the
lake speed, and its area in the lake dilution. river is the rule before, which
walked such a lake as river. It passed a pre-stated no-harm test, but the E. coli
data barely touch it (docs/LAKE-PATH-TEST-2026-10.md). Click-anywhere follows the
same mode.
Lakes. A click inside or within 150 m of a WFD lake polygon is treated as
that lake. The lake's centreline links (OS Open Rivers form = lake) inside the
polygon define its outlet; every overflow upstream of the outlet contributes,
its water routed along the network to where it enters the lake and then by the
shortest way through the WFD lake's water to the click, not a straight line over
land (lake distance water, the default after docs/LAKE-DISTANCE-2026-10.md;
DIPCAST_LAKE_DISTANCE_MODE=line is the rule before). At Pooley Bridge,
Glenridding WwTW's line ran 6.7 of 10.9 km over land; through the water it is
11.5 km. Lake dilution divides the weight by
1 + area / 5 km², so Windermere's north basin (8.7 km²) cuts a shoreline spill's
weight to about a third. Windermere is two WFD basins and is treated as such.
A WFD lake that an overflow's water crosses on the way to a spot, river or lake, is
a well-mixed lake if HydroLAKES gives it a residence time τ (282 of the 564). τ is
divided by R, the median ratio of spill-day to long-term mean flow at the nearest EA
flow gauge (lake path residence_spill, the default after
docs/LAKE-RESIDENCE-SPILL-2026-10.md; 10 lakes gauged, the rest take their median,
1.88). Then a share 1 / (1 + ln 10 · τ/R / T90) of a spill's bacteria leaves the lake
alive, after about 13 hours. So Grasmere (τ 39 days, R 2.24) passes 3%, and Grasmere
WwTW counts at Rydal Water at 0.0022, not 0.040. (Those figures are at T90 = 30 h; at
72 h, from 17 Oct 2026, Grasmere passes 7%, after about 29 hours, and Grasmere WwTW counts
at Rydal Water at 0.018: 0.19 under lake path lake, 0.25 under river, 0.31 at Grasmere itself.) A lake with no τ (Elter Water, most
under 10 ha) is crossed at 0.05 m/s with its own area term, as from 11 Oct 2026 (lake
path lake). A lake-form link outside every WFD polygon (a mill pond, a weir pool)
stays river. The spot's own lake keeps the fixed drift and the area term, and there
is no wind term. The volumes are modelled; full mixing and a median spill-day flow
both make a lake remove more of a spill than it may in a flood or in summer layering.
A click on the map that is not near a WFD lake falls back to the centreline
heuristic: a lake centreline within 1.5 km, if it is nearer than the nearest
river after a bias towards rivers.
A spot listed as a lake (kind = lake) is placed more strictly, because a lake
borrowed by mistake brings its overflows with it. Off a WFD shore, it goes into
a WFD lake within 1 km only if the spot is named after it, word for word
("Ullswater, Pooley Bridge" is 587 m from the Ullswater polygon; "Blea Tarn,
Langdale" does not take Little Langdale Tarn). Otherwise it is a small lake the
WFD set leaves out (under 50 ha), and its own centreline, if OS Open Rivers
draws one, must be within 500 m and not inside a WFD lake. Otherwise it is
isolated: no monitored overflow reaches it by water. Until 11 Oct 2026 any
centreline within 1.5 km counted, and Loughrigg Tarn took Grasmere's, 883 m
away, with Grasmere WwTW as its overflow upstream. No water flows from Grasmere
into the tarn (reported by a Lake District water-quality group, 11 Oct 2026).
Spot placement. A clicked point goes to the nearest link within 1.5 km. A
spot in spots.csv or spots-osm.csv also names its river (the river column, blank for lakes),
and goes to the nearest link carrying that name within 1 km, even when a link
of another name is nearer. OS Open Rivers' alternative name counts, and Welsh
names are read as English (network/names.py: Afon Gwy is the Wye, Afon Hafren
the Severn, Afon Tefeidiad the Teme). Before this, on 2 Oct 2026, six river
spots were traced up a side beck or a lake and showed no overflows upstream;
Crook o' Lune, for one, sat 262 m from Escow Beck and 672 m from the Lune. With
no such link the nearest is kept and the build logs a warning. Each build then
checks every river spot and adds one line to build.warnings in spots.json
listing any whose snapped watercourse shares no word with its river, that sits
more than 250 m from it, that has no river, that was placed as a lake or an
isolated lake, or that snapped to a canal (build_site.placement_check).
Qualifiers such as "Great", "West" and "and" do not count as shared words, so
the Great Ouse does not match Great Agill Beck. A spot moved
from a mapped side channel to the main one (adopted_main_channel) is judged
by name only, since its distance is the side channel's offset.
Right now. Same weights applied to live status. A spill counts in full from
its start until its water has passed the spot (its end plus its travel time; for
one still discharging, its travel time from now), then decays with T90, and stops
counting 48 h after its water passed. Until 4 Oct 2026 a finished spill decayed
from its end, whatever its travel time, so a spill 20 h upstream was let go while
its water was still arriving. Replayed over the 152 builds from 13 Sep to 4 Oct
2026 (the replay matched the published level in 99.6% of 9,101 spot-builds), the
change raised 55 spot headlines, at 15 spots, and lowered none. A spill whose
water has not reached the spot yet counts from its start: also waiting for the
water would have lowered 15 headlines while sewage was on its way, one to "Low
risk for the next five days" with the water 20 minutes off. The page says instead
when it arrives (now.arriving).
Skill with real forecasts. The table above uses reanalysis rainfall, so it
excludes weather-forecast error. scripts/verify_leads.py repeats the 2025
test with Open-Meteo's archived forecasts by lead time, using the same
held-out model (trained 2023-2024). Lead 0 is the latest run for the day
(what "today" uses); lead k is the forecast issued k days earlier.
| Rainfall source | Brier | AUC | Brier skill vs climatology |
|---|---|---|---|
| reanalysis (ERA5-Land) | 0.0447 | 0.929 | 0.33 |
| forecast, lead 0 (today) | 0.0429 | 0.933 | 0.36 |
| forecast, lead 1 (tomorrow) | 0.0469 | 0.913 | 0.30 |
| forecast, lead 2 | 0.0484 | 0.913 | 0.28 |
| forecast, lead 3 | 0.0504 | 0.902 | 0.25 |
| forecast, lead 4 | 0.0536 | 0.892 | 0.20 |
Today's forecast rain beats reanalysis, presumably because the forecast model
runs at about 2 km against ERA5-Land's 10 km. Skill decays with lead but stays
ahead of the rain rule (0.0621) four days out. The daily rainfall itself has a
mean absolute error of 2.0 mm at lead 0 rising to 3.1 mm at lead 4, and catches
67% of days over 10 mm at lead 0 against 45% at lead 4. Treating forecast rain
as certain makes longer leads over-confident (at lead 2 a 75% forecast verifies
at 65%), and the model is also over-confident at the top end even with today's
rain (an 85% forecast verifies at 82%, a 93% one at 86%), a shift between years
that a two-parameter Platt scaling cannot remove. So an isotonic map (monotone,
piecewise, about 50 knots) is fitted per lead on the held-out year's forecasts
and applied in forecast_point; data/processed/lead_calibration.json holds the knots,
with the Platt parameters kept alongside for comparison. After calibration every
lead-2 bin sits on the diagonal (the 75% bin verifies at 75%, the 82% bin at
82%) and lead-4 Brier improves from 0.0536 to 0.0522. The map is fitted on the
same 2025 data it is scored on, so treat those figures as slightly optimistic;
the live "By lead time" table shows raw and calibrated Brier side by side, which
is the check that the 2025 map still fits later years.
Validation against measured E. coli. The Environment Agency publishes weekly
lab samples at 38 inland designated bathing waters (20 rivers, 18 lakes). For
every sample from May 2023 to September 2026 (2,165 samples; 1,750 at the 32
sites with monitored overflows upstream) scripts/validate_ecoli.py computes
what SwimSignal would have said for that day from reanalysis rainfall, and
compares it with the naive competitor, rainfall at the site in the previous
48 hours. Results (Spearman rank correlation with log E. coli; AUC for samples
over 900 cfu/100 ml, the inland "sufficient" threshold):
17 Oct 2026: rerun with the fitted transport settings (0.2 m/s, T90 72 h,
square-root dilution; docs/TRANSPORT-FIT-TEST-2026-10.md), the spill risk scores
pooled ρ 0.57, within-site ρ 0.37 and AUC > 900 0.81 on 1,685 samples at 31 sites.
On rivers: within-site ρ 0.47 (rain in the last 48 h 0.475), AUC > 900 0.77. On
lakes: within-site ρ 0.29, AUC > 900 0.61. Pooled ρ falls: the new settings rank
days within a site better and sites against each other less well. Site-mean risk
still ranks the sites at ρ = 0.76.
12 Oct 2026: the figures in this section are from 14 Sep. Rerun on the network
after #184, the refilled rain cache and the site's own placement of each bathing
water (#199; docs/ECOLI-REFIT-SITE-PINS-2026-10.md), the spill risk scores
pooled ρ 0.63, within-site ρ 0.31 and AUC > 900 0.81 on 1,684 samples at 31
sites (Colwick West Lake has no overflows upstream with the site's placement).
Site-mean risk ranks the sites at ρ = 0.76; site-mean rainfall gets −0.23. The
current figures are in data/processed/ecoli_validation*.json and on the
Accuracy page.
| Predictor | Pooled ρ | Within-site ρ | AUC > 900 |
|---|---|---|---|
| rainfall, previous 48 h at the site | 0.09 | 0.35 | 0.65 |
| SwimSignal spill risk (rain-driven spill model through transport) | 0.57 | 0.29 | 0.80 |
Two different questions hide in that table. Which sites are contaminated:
site-mean SwimSignal risk ranks the 32 sites' mean E. coli at ρ = 0.68, and
site-mean rainfall does not (−0.25). The transport layer, the part of SwimSignal
that is new, is what carries this. Which days are bad at a given site: on the
raw scale rain in the last 48 hours leads (0.35 against 0.29; on rivers 0.48
against 0.41), but a within-site comparison depends on the scale the site mean
is removed on. On the logit scale, the scale SwimSignal combines contributions on,
spill risk is level with rain (0.35 against 0.35; rivers 0.47 against 0.48).
Leave-one-year-out linear fits on that scale
(scripts/validate_ecoli_combined.py) give within-site ρ of 0.35 for rain
alone, 0.36 for spill risk alone, 0.37 for both and 0.39 with season added
(rivers 0.50, 0.51, 0.53, 0.53), so the two carry partly different
information but neither explains most of the day-to-day variation. At the nine
United Utilities sites, where actual spill events are known (601 samples, 542
of them on lakes), routing the real spills through the transport step
correlates with E. coli at only 0.23 within site, and a spill had reached the
spot within the previous 48 hours for 29% of samples: at those lakes,
bacterial spikes are mostly not overflow-driven.
The honest reading: SwimSignal's forecast tells a swimmer how exposed a spot is
and when the overflows above it are likely to spill, and on rivers its
day-to-day signal is as good as recent rainfall; it does not yet capture the
diffuse runoff (farms, roads, urban drainage) that rain washes into rivers
regardless of overflows, and at inland bathing waters neither signal captures
most of the day-to-day variation. The next model should predict E. coli
exceedance directly from both, which these 2,165 samples make possible. Full
tables: data/processed/ecoli_validation*.json|csv.
Correction (14 Sep 2026): the first run of this validation (12 Sep) reported a within-site ρ of 0.23 for SwimSignal. That run fed only the sample days into the travel-time shift, which assumes consecutive days, so a fifth of each overflow's contribution landed on the following week's sample; it also applied the lead-4 rather than lead-0 Platt calibration. The hindcast now runs on a continuous daily grid and the figures above are from the corrected run. The observed-spill result was computed differently and did not change.
Would forecast-led sampling catch more? A question for anyone who pays for
water samples: if you could only afford half your sampling days, would choosing
them from the rain forecast catch more of the failures? scripts/sampling_plan_test.py
ranks each site's sampled days by rain in the previous 48 h and keeps the wettest
half. On rivers (907 sampled days in 2024-26, 196 exceedances of 900) that keeps
76% of the exceedances with rain as it fell, 75% with the forecast issued that
day, 70% with the forecast issued the day before, and 69% with the forecast
issued two or four days earlier; a fixed schedule keeps 50%. A "sample only if
more than 5 mm is forecast" rule, decided two days ahead, keeps a quarter of the
days and half the exceedances (43% exceedance rate on the days it picks, 14% on
those it skips). Most of the skill is lost between same-day and one-day-ahead
decisions, not after, so a plan made four days out is about as good as one made
the day before. It is the rain forecast doing this work, not the transport
layer; on lakes (16 exceedances) there is nothing to plan around. Archived
forecasts by lead come from scripts/fetch_rain_leads_bathing.py; results in
data/processed/sampling_plan_test.json. The test can only choose among days the
EA happened to sample, so it measures ranking skill, not the value of sampling on
days nobody did.
Correction (28 Sep 2026): the first run (15 Sep) summed each 48 h window with
pandas, which reads an all-missing window as 0 mm, and Open-Meteo's archive has
no lead 1-4 rain for 2023, so those samples counted as dry at every lead but the
same day. It now uses the production window rule (ecoli.rain_windows: 90% of
the hours must carry a value) and compares every lead on the same 2024-26
samples. The river figures moved by one to two points (lead 2 from 68% to 69%;
the same-day figure stayed at 75%); the conclusions did not change.
E. coli exceedance model. The map's "E. coli > 900" column: the estimated
probability that a midday sample exceeds 900 cfu/100 ml. scripts/train_ecoli.py
fits a logistic regression from things SwimSignal can compute anywhere: rain at the
spot in the previous 48 and 24 h, SwimSignal's overflow exposure for the day, lake or
river, and season. The rain is Open-Meteo's archived lead-0 forecast, the same
source the map uses for "today". The competitor that matters is rain alone: the
question is whether the overflow exposure adds anything a rain gauge would not.
Refitted 17 Oct 2026 on the fitted transport settings
(docs/TRANSPORT-FIT-TEST-2026-10.md): 1,502 samples at 31 bathing waters, as before.
Leaving a year out: Brier 0.092, AUC 0.82 (rivers 0.76, lakes 0.54); rain only
0.096 and 0.79. On rivers the gain over rain alone is +7.1 thousandths with a year
left out (95% interval by site-weeks [+1.3, +12.6]) and +6.8 with a site left out
([+1.2, +11.9]); by calendar week the year-out interval still includes no gain
([−1.4, +16.6]). Against rain + season it now holds under all three resamplings,
with a year or a site left out. Forward in time: +12.8 [+6.0, +19.6] by site-weeks.
The exposure coefficient rose from 0.59 to 0.79 per SD.
Refitted 12 Oct 2026 on the site's placement of each bathing water (#199;
docs/ECOLI-REFIT-SITE-PINS-2026-10.md): 1,502 samples at 31 bathing waters.
Leaving a year out: Brier 0.094, AUC 0.81 (rivers 0.75, lakes 0.53); rain only
0.096 and 0.79. On rivers the gain over rain alone shrank (leave one year out,
+3.9 thousandths). With a year or a site left out, its 95% interval now includes
no gain under all three resamplings. Against rain + season it holds under all
three with a year left out, not with a site left out. Forward in time it is much
as before (+8.1 [+2.6, +13.4] by site-weeks). The tables below are the 16-17 Sep
fit; the current ones are in data/processed/ecoli_model_eval.json and on the
Accuracy page.
Rebuilt on 16-17 Sep 2026 after the rain-window fixes: 1,548 samples at 32 bathing waters, 2024-2026 (the 202 samples from 2023 are gone, because Open-Meteo's previous-runs archive has no lead-1 to lead-4 rain for 2023 and the old code had been summing those nulls to 0 mm and scoring them as dry). Three tests, each scored on data the model never saw:
| Test | Model | Brier | AUC | Rivers: Brier / AUC | Lakes: Brier / AUC |
|---|---|---|---|---|---|
| leave one year out | climatology by type | 0.116 | 0.62 | 0.181 / 0.35 | 0.025 / 0.38 |
| rain only | 0.093 | 0.80 | 0.142 / 0.71 | 0.025 / 0.35 | |
| rain + season | 0.094 | 0.80 | 0.143 / 0.73 | 0.025 / 0.37 | |
| spill exposure only | 0.097 | 0.80 | 0.147 / 0.72 | 0.026 / 0.51 | |
| rain + exposure + season (the map) | 0.091 | 0.82 | 0.138 / 0.75 | 0.025 / 0.47 | |
| leave one site out | rain only | 0.092 | 0.79 | 0.140 / 0.72 | 0.025 / 0.06 |
| rain + exposure + season | 0.090 | 0.81 | 0.136 / 0.76 | 0.026 / 0.18 | |
| forward: fit 2024, score 2025-26 | rain only | 0.087 | 0.79 | 0.125 / 0.77 | 0.031 / 0.52 |
| rain + season | 0.087 | 0.79 | 0.125 / 0.77 | 0.031 / 0.52 | |
| rain + exposure + season | 0.083 | 0.81 | 0.117 / 0.81 | 0.031 / 0.53 |
The forward test uses the overflow exposure from the spill model fitted on 2023-24 only, without the 2025-fitted calibration map, so nothing downstream of the cut-off saw the test years. The season term on its own adds nothing (rain + season is level with or slightly worse than rain only), so the gain of the full model is the exposure term.
Is the gain over rain alone real? Cluster bootstraps on the out-of-sample river predictions under three dependence assumptions: resampling site-weeks (samples at one site in one week share weather and water body), whole sites (every week at a site moves together; 20 clusters), and calendar weeks across all sites (one storm hits many sites at once; 40-62 clusters, the most demanding). Brier gain in thousandths, 95% intervals, rivers (n = 907; forward n = 617):
| Test | Reference | by site-weeks | by whole sites | by calendar weeks |
|---|---|---|---|---|
| leave one year out | rain only | +4.6 [+0.3, +8.8], P 0.02 | +4.6 [+0.9, +8.7], P 0.01 | +4.6 [−2.0, +11.6], P 0.09 |
| leave one site out | rain only | +3.8 [−0.0, +7.7], P 0.03 | +3.8 [−0.5, +8.2], P 0.04 | +3.8 [−1.1, +9.1], P 0.07 |
| forward in time | rain only | +8.3 [+3.1, +13.2], P 0.001 | +8.3 [+1.8, +15.3], P 0.005 | +8.3 [−0.4, +17.7], P 0.03 |
| leave one year out | rain + season | +5.6 [+2.3, +9.1], P 0.001 | +5.6 [+2.2, +9.3], P <0.001 | +5.6 [+1.9, +9.4], P 0.001 |
| leave one site out | rain + season | +4.4 [+1.1, +7.9], P 0.004 | +4.4 [+0.6, +8.6], P 0.01 | +4.4 [+0.9, +8.1], P 0.007 |
P is the share of resamples in which the full model was no better. Against rain alone the gain is small (3-6% of the Brier score, 0.03-0.04 AUC) and survives resampling by site but not, at conventional confidence, resampling by storm week; the forward test is the strongest. Against rain + season, which isolates the exposure term, the gain holds under every grouping. On lakes there is nothing under any grouping, and the column is not shown for lakes on the map. Rain windows: 100% of the accepted 48 h windows at lead 0 are complete, so the 90% rule changed no total in the fitting data; at leads 1-4, 88.5% are.
Would the site have shown a figure? Under the production rule (a figure is withheld when more than 10% of transport weight comes from overflow-days without rain data) the replayed pipeline would have shown one on 100% of sample days at lead 0, 93% at lead 1 and 88.5% at leads 2-4; on wet days (48 h rain over 10 mm) 95% and 90%, and on exceedance days 97% at every lead from 1 to 4. So the abstentions fall mostly on dry, clean days, not the hard ones; the gaps are Open-Meteo previous-runs outages, not the model declining.
By lead, replayed. The earlier lead-time figures changed only the spot rain
and kept the lead-0 exposure at every lead. scripts/replay_ecoli_leads.py now
recomputes the exposure from the rain forecast issued k days earlier through the
spill and transport models, as the live site does. Leave-one-year-out Brier for
the full model against rain only, all sites (rivers in brackets):
| Lead | replayed exposure | rain only | fixed lead-0 exposure (old test) |
|---|---|---|---|
| 0 | 0.0908 (0.138) | 0.0934 (0.142) | 0.0908 (0.138) |
| 1 | 0.0972 (0.148) | 0.1012 (0.155) | 0.0952 (0.145) |
| 2 | 0.0987 (0.151) | 0.1022 (0.157) | 0.0957 (0.146) |
| 3 | 0.1024 (0.157) | 0.1065 (0.164) | 0.0984 (0.150) |
| 4 | 0.1000 (0.153) | 0.1045 (0.161) | 0.0960 (0.146) |
The old test was optimistic by about 0.004 at four days; the full model still
beats rain alone at every lead. The replay applies production's missing-data
rule (at most 10% of transport weight without rain data) rather than rejecting
any missing value, so its availability matches what the site would show; its
remaining approximations are the analysis series for the 30-day and antecedent
features and the stitched previous-runs fields. Reliability is close to the
diagonal below 0.2 and above 0.4; the 0.2-0.4 bins over-forecast (forecast 0.25
and 0.35, observed 0.21 and 0.20, on 179 samples). Coefficients are stored as JSON
(data/processed/ecoli_model.json); dipcast/model/ecoli.py computes the
features live from the spot's own rainfall cell, with the rain window ending at
midday to match when the EA samples.
Live scoring of the E. coli column. From 15 September 2026 every spot's daily
exceedance forecast is logged with the rain it used, and once a day the build
fetches this season's EA samples at the 38 designated bathing waters into the
state directory. Each sample is matched, per lead, to the latest forecast for
that spot and day that was issued before the sample was taken (a forecast made
at 23:00 is not a forecast of an 11:00 sample), and scored the same way as the
spill forecasts: Brier and AUC against the training-period exceedance rate for
rivers and lakes, same-day and in-advance leads reported separately, with the
most recent samples listed against what the map said. It appears on the
verification page as results arrive, usually within a week of sampling. The 2026
season ends in September, so the first real read of this table is next May.
Samples come from two EA services. The bathing-water service is asked first,
since it publishes first, but from 28 Sep 2026 its gateway refuses GitHub's
runners (HTTP 403 whatever the User-Agent), so the build stops at the first
refusal. The Water Quality Archive holds the same results about 3-7 days later
and answers the runners: on 28 Sep it had 675 of the service's 702 samples as
statutory monitoring, all with identical counts. Each bathing water's archive
point is recorded as wqa_point in data/raw/bathing_waters_inland.json by
scripts/map_bathing_waters_wqa.py, which matches samples, not just distance. A
sample found in both is taken from the bathing-water service. Each fetch attempt
is recorded (ecoli_samples_status.json in the state directory), and a site no
source answered for keeps its earlier samples. If no source answers at all, the
build log, the Actions run page (as an annotation) and the verification page say
so, instead of showing zero scores as if nothing had been sampled yet.
Beside the Environment Agency's daily risk prediction (3 Oct 2026). The EA
publishes a prediction for each designated bathing water every day of the season,
"normal" or "increased" (stp-risk-prediction.json?predictedOn=<day>); where it runs
a pollution risk forecast (PRF) for the site, prfOriginType is PRF_PROVIDED.
scripts/compare_prf.py fetched all 139 days of the 2026 season (15 May to 30 Sep)
and found that none of the 38 inland bathing waters had a PRF on any day; 34 appear
in the predictions at all, and there "increased" follows a posted notice (a
pollution incident, harmful algae), not a forecast. It scores the prediction in
force when each EA sample was taken, read as 1 or 0, beside SwimSignal's latest E.
coli estimate issued before the sample. The estimate has been logged since 15 Sep,
so the two meet on 30 samples at 22 sites (16-28 Sep), 2 of them over 900: Brier
0.072 for SwimSignal, 0.133 for the EA, 0.079 for the long-run rate, and neither
warned before either exceedance. Too few to judge. Over the whole season the
prediction was "increased" before 13 of the 598 samples it covered (1 over 900) and
"normal" before 44 of the 45 over 900. The EA's gateway refuses GitHub's runners, so
the script runs by hand and data/processed/prf_comparison.json is committed; from a
home connection it refused twice, after 55 and 62 requests, and answered again within
90 s, so the script pauses 2 s between requests, retries a refusal once after 90 s,
and caches each day's response under DIPCAST_CACHE/ea_prf/.
Live scoring rules for the spill forecasts (16-17 Sep 2026). The forecast
scored for each overflow and day is the latest one issued by 08:00 local time on
the issue day, so "today" is the forecast a swimmer had at breakfast, not the
end-of-day estimate the earlier rule (latest issue of the day) produced. Issue
days with no forecast by 08:00 are missed deadlines: counted and listed on the
verification page, not scored. An overflow-day counts as "no spill" only if the
poller recorded that overflow with a known status at least six times that day,
from a feed whose freshest LastUpdated was under 6 h old (a feed with no
LastUpdated at all is not current; see the correction of 2 Oct below), with no unobserved
stretch longer than 8 h (6 h until 1 Oct 2026; see the correction below;
counting midnight to the first poll and the last poll to midnight), and once
the next day. live_coverage.parquet holds one row per
overflow per day: known, unknown and stale poll counts and a 48-bit mask of the
half-hour slots observed, from which the scorer derives first and last
observation and the longest gap; a repeated poll in the same slot adds nothing.
poll_log.parquet records each poll's per-company row count and feed age, so
outages and stale feeds are visible (six companies re-stamp every record each
refresh; Northumbrian and Southern stamp a record only when it changes; South
West Water publishes no stamp). Before this, any polling at all on a day counted
as coverage for every overflow, and the history file, which keeps one row per
distinct status, could not say which overflows had actually been seen. Scores
from before the rule change are withdrawn; the table restarts as coverage
accumulates, and the page reports the same scores under a stricter 6 h gap rule
alongside. The withdrawn scores are kept in
data/processed/verification_live_oldrule_2026-09-28.json (49,380
forecasts for 9,876 overflow-days at five leads, 17-27 Sep 2026, a dry spell). They were not good: forecasts
averaged 2.7% against 1.3% observed, forecasts between 10% and 70% verified at
a third to a half of their stated value, and only at United Utilities, the one
company in the training data, did they beat a flat forecast at the period's
own spill rate. The climatology baseline divides each overflow's annual spill count
by 365, an approximation: the returns count spills by the 12/24-hour block
method, and scripts/spill_day_ratio.py finds 1.00 spill-days per counted
spill pooled over 5,886 United Utilities site-years, which supports the
approximation there without establishing it per overflow or company.
Correction (29 Sep 2026): these rules were written on 16-17 Sep but reached the
live site on 28 Sep, when the commit carrying them was pushed to main (16:51
UTC). The coverage file shows it: no overflow-day row from 16-27 Sep carries a
half-hour observation mask, and 98% of 28 Sep's rows do. Under the gap rule a
day without a mask can never be scored, so the live table was empty on 29 Sep
(0 of 70,663 candidate overflow-days); 29 Sep is the first day that can be
scored, if that day's polls meet the rule. The
scorer now reports the first masked day (observations_from) and the
verification page says why the table is empty.
Correction (1 Oct 2026): the 6 h gap rule assumed the poller ran every 30
minutes, as the workflow asks. GitHub starts the schedule when it can: on 29
and 30 Sep it ran 4 and 5 times, with daytime gaps of 6.4-6.8 h (03:25, 09:48,
16:15, 21:05 on the 29th), so every one of the roughly 14,000 masked
overflow-days failed the gap test and nothing was scored (0 of 80,753
candidates on 1 Oct; the other steps passed about 5,040 a day). Over 17-30 Sep
the longest daily gap was under 6 h on 10 days of 14 and under 8 h on all 14,
so at 6 h about a day in three would have been lost for good, and the first
three masked days all were. From 1 Oct the limit is 8 h, which would have
scored 4,893 overflow-days on 29 Sep and 4,936 on 30 Sep, and the stricter
rule reported alongside is the old 6 h, so the effect of the change stays
visible. The next-day rule still catches a single event that ends late; what
8 h gives up is a second event that starts and ends unseen inside one stretch.
Established by running the scorer's own functions on the state release of
1 Oct 10:32 UTC.
Correction (2 Oct 2026): South West Water's feed carries no StatusStart,
LatestEventStart, LatestEventEnd or LastUpdated on any row. Three things
followed. A spill was found only from the event fields, so none of its
overflows ever counted as spilling, although the history holds rows marked
discharging (74 overflows on 29 Sep, 165 on 30 Sep): its 135 scored
overflow-days all read "no spill", base rate 0, skill -0.52. The history's
de-duplication key (site, status, status start) was the same on every poll, so
each overflow kept one row per status and a later day overwrote an earlier
one. And the feed-age test read a missing stamp as current. Now: a row marked
discharging with no event times is a spill on the local day of that poll; a row
with no status start is kept once per local day polled, at every company.
Anglian, United Utilities and Wessex have such rows too (294 in the 2 Oct 14:08
snapshot, 18 of them with event times), so the history grows by about 300 more
rows a day, and an event the old key would have overwritten is now kept: their
spill-day counts can only rise from now on. On the 2 Oct history, which the old
key had already de-duplicated, no other company's count changed; and a feed with no
LastUpdated is never current, so South West Water's overflows were not scored
at all until the site read its camelCase lastUpdated on 4 Oct; they are scored
from 5 Oct (unstamped_feed_days applies this to the days polled before the
change, from poll_log.parquet, ignoring rows from before the log recorded feed
ages on 28 Sep 16:51 UTC, when every company's age reads as missing). The feed sees only what is discharging at the
moment of a poll, a few times a day, so a day of "not discharging" polls cannot
support "no spill". Its scores are withdrawn: 14,643 of the 14,778 overflow-days
scored on 2 Oct remain, by the scorer run on the state release of 2 Oct
14:08 UTC.
The scorer also reports why each unscored overflow-day was not scored
(uncovered_by_reason: before observations, feed not current, gap, too few
polls, no poll the next day), and a scoring_window from observations_from
to yesterday. Days before the masks existed cannot be scored, so they are
counted apart (n_before_window) rather than in the window's candidates. On
2 Oct's state, 60,300 of the 65,805 unscored overflow-days were before
observations, 5,325 failed the gap rule (5,025 of them on 28 Sep, when the
masks began in the afternoon) and 180 were South West Water's.
Warnings, misses and false alarms (4 Oct 2026). Brier scores and AUC
mean nothing to a swimmer, so the scorer also counts both forecasts as
warnings (warning_table in verification.json, from
forecast_log.warning_counts). The line is the site's own High risk, where
alerts are sent: an overflow's calibrated spill chance of 40% or more
(SPILL_WARN_AT; a spot beside that overflow would read High risk), and an
E. coli estimate of 25% or more (ECOLI_WARN_AT), on rivers only, one row per
sample with the latest estimate issued before it. Each forecast is a hit, a
miss, a false alarm or a correct quiet day, and the table gives the share of
events warned of, the share of warnings that came true and the share right,
always beside the share that saying "no" every time gets right. warning_lift
is how many times likelier the event was after a warning than across all
forecasts (the share of warnings that came true over the base rate). On the live
scores of 4 Oct 00:15 (23,395 overflow-day forecasts, 29 Sep to 3 Oct) the
forecast warned in 152 of the 725 forecasts for an overflow-day that spilled
(21%; 145 such days, each forecast at five leads), 152 of its 363 warnings came
true (42%, against a 3.1% spill rate across all forecasts: 13.5 times as
likely), and it was right on 96.6% against 96.9% for never warning.
The E. coli counts were 15 river samples, 1 over 900, no hits, 1 miss and 2
false alarms: too few to judge (under 100 samples or 10 exceedances).
Service record (4 Oct 2026). forecast_log.service_record writes a
service block into verification.json from the pipeline's own logs. Runs
come from poll_log.parquet, which every run writes once (whole days from
the day after its first poll, 16 Sep 2026, to yesterday), against the 48 a
day that the cron in site.yml asks for; a test checks the two agree. Feed
outages are the polls in which a company returned no rows, and the morning
forecast is the days with a spot forecast issued by 08:00. A run log,
build_runs in dipcast.duckdb, has one row per build from 4 Oct 2026
(log_build, called by scripts/build_site.py before it writes
verification.json, and on BuildUnhealthy): spots tried and forecast, the
feeds that returned nothing, and what started the run. It lives in the same
file as the forecast log, so it survives between runs the same way (the
Actions cache, then the state release) with no change to the workflow. On
the state release of 3 Oct 20:06 UTC: 148 runs on 17 whole days, a median of
7 a day, 18% of the 816 asked for; the longest wait 7.7 h; a forecast out by
08:00 on all 17 days; one empty poll, Yorkshire Water's at 00:18 on 2 Oct.
A company feed that fails, or returns no rows, keeps its last snapshot in
live_latest.parquet with status -3 (feed down) and the time it last answered;
a spot's "Right now" tile names the company and that time, and counts its
overflows as not reporting. The build warns when a company returns no rows. It
still publishes when none does, with every overflow marked feed down: not
publishing would also freeze the rain forecasts and leave the old statuses on
the page with no note.
A short read is treated the same way (6 Oct 2026). Yorkshire Water's layer is rewritten whole on each refresh, and a read during a rewrite can return part of it: 1,000 of 2,179 rows on 16 Sep, 1,179 on 6 Oct at 13:48 UTC. That build listed 1,000 overflows as "not in the company's live feed", though all were in it. Now a read with under 90% of a company's usual rows (the median of its last 10 polls that returned any) is read again, up to three reads. Overflows still missing keep their last snapshot as feed down, and the build warns. The scorer was never affected: a missing overflow gets no observation that poll, so its day cannot pass the coverage rule as dry.
Data state of each overflow (4 Oct 2026). A frozen feed must not read as
"not discharging". Each overflow now carries data_state beside status
(ingest.live.data_states), in the overflow table, the contributors in
spots.json and the features of overflows.geojson. status keeps its numbers.
data_state | When | The spot page says |
|---|---|---|
live | status 1 or 0 from a feed whose freshest record stamp is under 6 h old (FEED_CURRENT_H, the scorer's test) | "discharging", "not discharging" |
stale | status 1 or 0 from a feed whose freshest stamp is older, or that gives no time | "no update since 2 Oct, 14:00", "no update time from the company", "discharging at its last update, 2 Oct, 14:00" |
offline | status -1 (monitor offline), -3 (company feed down), or -2 at a company that has a live feed | "monitor offline", "company feed down", "not in the company's live feed" |
no_feed | status -2 at a company with no live feed SwimSignal reads (Dŵr Cymru Welsh Water, whose layer has no licence yet) | "no live feed" |
A company whose records carry no stamp at all counts as current when its layer
was written within 6 h: fetch_live then reads the layer's own last data edit
(editingInfo.dataLastEditDate from <layer>?f=json, one more small request
a poll). The scorer does not use that time: a current snapshot says nothing
of what happened between polls. In the "Right now" tile an overflow reports
live only if it is discharging, stopped lately and still counted, or quiet on
a current feed (monitored_upstream). A stale feed is named with its last
update, as a feed that is down is (feed_stale, stale_upstream), and its
quiet overflows are drawn in the grey of offline. Before this, every overflow
listed in a live feed counted as reporting, so a monitor its company marked
offline read as quiet: at River Ribble, Stainforth Force, on the 4 Oct 00:16
build, the one overflow upstream was offline and the tile said "1 of 1 report
live". The risk figure is unchanged: an overflow whose reading is unknown adds
nothing to it, as before.
Freshness of each record and each reading (11 Oct 2026). An audit of the state of 11 Oct 18:16 UTC found three gaps; each is closed here.
ingest.live.stamps_every_record; on 11 Oct: Anglian, Severn Trent,
South West, Thames, United Utilities, Wessex and Yorkshire at 97.9-100%,
Northumbrian 1.4%, Southern 0.3%). There, a record whose own stamp is missing
or over 6 h old is stale with record_stale true and its own stamp as
feed_updated_at, and the scorer counts its polls stale too (frozen_records,
in data_states and coverage_rows). The tile names it apart from a stale
feed: "Wessex Water has not updated 2 of its overflows here since 31 Mar,
15:23". Northumbrian and Southern stamp a record only when it changes, so for
them an old stamp says nothing and the company test alone applies. A company
between half and 95% is logged and read as stamp-on-change.carry_forward now keeps last_status, and
transport.running counts a carried row whose last status was 1 as running:
in the risk, in "discharging", and on the page as "discharging when the feed
last answered".data_state is decided when the
build polls. now.quiet_by_feed gives each company's quiet overflows and
their feed time, and the page stops counting them as reporting live once that
time is over 6 h old where it is read (ageReadings in index.html), well
before the 8 h "Stale" notice.The spill model's covariates (lta_spills, spill_hours,
edm_operational_pct) were filled, where an overflow has none, over the
overflows upstream of one spot at a time, so the same overflow took a different
value at each spot. overflows.build_overflows now fills them once, from
company-wide medians over the whole table, into <covariate>_filled
(features.fill_covariates); the raw columns, which the pages show, are left
empty. The accuracy baseline reads the same filled count.
Live feed fields. From each layer's own description (<layer>?f=json,
read once per layer at 23:29 UTC on 3 Oct 2026, no data query; trimmed into
tests/fixtures/live_layer_fields.json). All nine are the National Storm
Overflow Hub's schema and nothing more: Id, Company, Status,
StatusStart, LatestEventStart, LatestEventEnd, Latitude, Longitude,
ReceivingWaterCourse, LastUpdated, and an object id. Status is a coded
value in all nine: 1 "Start" (discharging), 0 "Stop", -1 "Offline". No layer
has a field for a record's validation or verification, a dry-weather spill or
a high river. The competitor review of 3 Oct named such flags at United
Utilities, Thames Water, Northumbrian Water and Severn Trent; they are not in
the feeds SwimSignal reads, and where those companies publish them was not
checked, so the page cannot show them.
| Company | Field names | Record stamps (feed_age_hours, from the poll log) | Layer data last written, at the read |
|---|---|---|---|
| Anglian Water | the hub's | every record, each refresh | 23:26 UTC |
| Northumbrian Water | the hub's | a record only when it changes | 23:19 |
| Severn Trent Water | the hub's | every record, each refresh | 23:19 |
| South West Water | the hub's in camelCase, all but Id (status, statusStart, lastUpdated, ...) | none read so far (below) | 23:20, schema rewritten at the same time |
| Southern Water | the hub's | a record only when it changes | 20:58 |
| Thames Water | the hub's | every record, each refresh | 23:23 |
| United Utilities | the hub's | every record, each refresh | 23:03 |
| Wessex Water | the hub's | every record, each refresh | 23:23 |
| Yorkshire Water | the hub's | every record, each refresh | 23:18 |
Until 4 Oct fetch_live matched field names exactly, so it never read South
West Water's statusStart, latestEventStart, latestEventEnd,
lastUpdated, receivingWaterCourse, latitude or longitude, whatever
they held; its locations came from the point geometry. It now reads field
names whatever their case (hub_names). Whether those fields hold values was
not checked, since no data query was made. The 2 Oct correction above found
no LastUpdated or event times on any South West Water row; a check of
live_latest.parquet, which this code wrote, could not have seen them. If
they do hold values, South West Water's overflows get record stamps and event
times from the next poll, and the scorer will score their days when the rules
above are met.
Algae (an observation, not a forecast; 28 Sep 2026). At every sampling visit to a
bathing water the EA sampler records one of four levels of algae: none, a trace (1-2
items), some at intervals (3-6), or enough to be objectionable (more than 6). This is
"Bathing Water Profile : Algal Bloom", determinand 4824 in the Water Quality Archive,
recorded against the same sample as the E. coli result. On 28 Sep the archive held 2,925 of
these checks at the 38 sites since 2020. From 2023, algae was seen at 310 of 1,304 lake
checks (63 objectionable) and 64 of 1,012 river checks (11). 58 of the 63 objectionable
lake checks were at four sites: the Serpentine (21), Colwick (15), Frensham Great Pond (12)
and Hampstead's Ladies' Pond (10).
There are no chlorophyll, cyanobacteria-count or toxin results at these sites, and the
check does not tell blue-green algae (which can be toxic) from harmless kinds. Too few
objectionable checks to fit a model, so the page shows each bathing water's latest check,
dated, and the season's tally. dipcast/algae.py fetches the season (from 1 May) once a
day into algae_checks.parquet in the state directory. A failed or empty fetch keeps the
checks already held, and nothing about it stops a build. The file is not in the state
release, because the archive can refill it on any build. Where it adds something: the
Serpentine has no monitored overflow upstream, so the forecast has nothing to flag, and
in 2026 the sampler saw algae there at 18 of 20 visits, 12 of them objectionable.
Versions. Every logged forecast carries a stamp of the spill model,
calibration map and E. coli model (content hashes), the code version and git
sha, and the weather source (forecast.model_version). Live scores are broken
down by stamp on the verification page, so a change starts a new row rather
than being averaged into the old one.
Level checks (28 Sep 2026). Beating climatology says little in a dry
spell, because climatology knows nothing about the weather. The live table
therefore also reports the mean forecast against the observed spill rate,
overall and per company, and the score of a flat forecast at the period's own
spill rate. That flat forecast uses hindsight, so it is not a rival, but a
forecast that scores worse than it is pitched at the wrong level. From 28 Sep
each logged overflow-day forecast also carries the target-day rain it assumed
(rain_mm), and the page groups scores by it: excess on forecast-dry days
means the model's floor is too high, excess only on wet days points at the
rain forecast or the model's response to rain. A seasonal climatology was
considered and dropped: United Utilities' 2023-25 events put September at or
above the annual average (monthly factors 1.04, 1.06 and 1.91), so it would not
have made a dry September harder to beat.
Missing rainfall is unknown, not dry (16-17 Sep 2026). The E. coli rain windows
used to count timestamps rather than finite values, so a null-filled forecast
passed the completeness check and summed to 0 mm; the daily spill features had
the same failure. Both now require 90% (windows) or 20 of 24 (days) finite hours
and return NaN otherwise. The window is (t − 48 h, t]: exactly 48 hour stamps,
each the rain in the hour ending at that stamp (until 17 Sep it summed 49). An
accepted window with a few missing hours still sums only the hours it has, so
the finite fraction is kept (rain_48h_coverage in the API) and the evaluation
reports the gain on complete windows only alongside all accepted ones. Further, the transport step reports the share of weight arriving
from days without rain data, a day with more than 10% missing gets no figure on
the map ("no data"), overflow-days forecast without rain are excluded from live
scoring, and a build in which most forecasts fail or lack today's rain exits
non-zero so the previous site stays up. The live forecast's daily grid now starts
history_days(travel) days back, the same rule as the hindcast, so a 48 h travel
time contributes to today (it started at yesterday before, and did not).
verify_leads.py also reports a cross-fitted score (each month
calibrated by a map fitted on the year's other months) as the honest estimate
of what it does for an unseen day.bwq_samples.parquet fall in May-September), so then nothing tests it. It
is still shown then, marked † on the page, but since 1 Oct 2026 it does not
count towards the level out of season: on that day it alone had put 35 of 89
spots on high or very high while the spill forecast read low at 78 of 87,
which would have kept the list red all winter on an untested figure. The
spill forecast still answers to rain, so a wet autumn day still turns amber
where the overflows upstream are likely to spill. Its day-of-year term is held at
30 September's value from October to January and 1 May's from February to
April (ecoli.season_day): left to run, the fitted curve, with rain and
exposure fixed, rose from 38% on 30 September to 48% in mid-December with
nothing to check it, and the term adds no skill even in season (rain + season
scores level with rain alone). The fitting samples all fall inside the held
range, so the fitted model is unchanged.adopted_main_channel in the API.
Until 11 Oct 2026 62 of those joins (653 then) ran from a dead end to the head
of a link flowing into it, making a loop; at the Hampshire Avon by Ratfyn the
loop drained nowhere. Such joins are now left out, and put back (6 of them)
only where no rule mends the dead end and the loop has a way out to the sea.
The network is mended further as it loads (): 59 links
drawn into a dead end are turned round where the river carries on from their
other end under the same name, 28 dead ends are joined to the nearest node of
their river's name within 600 m, and lists by hand
what no rule finds (the Test's braid above Chilbolton; The Gog at All Cannings, drawn
backwards into a dead end with no other link, turned round on 12 Oct 2026). Each rule looks only at
dead ends with 10 km or more of network upstream.
lists the dead ends and loops before and after: of 175 with 30 km or more
upstream, 36 were links drawn the wrong way, 16 gaps, 71 side arms whose river
carries on without them, 15 lakes with no outflow or canal ends, and 37
unclear, left alone.Working end to end on a local machine: click anywhere on a river or lake in England and get a "right now" risk from live status plus a five-day forecast, with the contributing overflows listed and drawn on the map. Verified on a held-out year (table above). Unit tests (run by the site workflow before every build) cover the label exploder, rainfall features, the risk combination, missing-rain handling, the travel-time history window, the decision-time and coverage rules of the live scorer, the issued-before-sample rule of the E. coli scorer, coverage accumulation in the poller, the build-health guard, and the EA sample fetch (the archive stands in when the bathing-water service refuses, a total failure is flagged, a partial one keeps earlier samples, requests carry the contact User-Agent).
Done since v1: reliability release of 16-17 Sep (missing rain is unknown not dry; strict 08:00 headline with missed deadlines reported; per-overflow observation masks with a longest-gap rule and feed-freshness check; version stamps on every forecast with live scores by version; 48-stamp rain windows with coverage kept; bootstrap intervals by site and by storm week and the rain + season comparison; replay under the production missing-data rule with availability reported; decision-time and coverage-gated live scoring; issued-before-sample rule for the E. coli scorer; travel-time history window shared with the hindcast; build-health guard; leave-one-site-out, forward-in-time and bootstrap tests of the E. coli model with the lead-time exposure replayed; exposure shown as a 0-100 index, E. coli column withheld on lakes and flagged out of season; stale-forecast banner); Southern Water live feed (all nine English companies now live); E. coli exceedance model on the map, logged and scored live against new EA samples (15 Sep); per-lead isotonic calibration replacing Platt (15 Sep); second-pass outfall snapping (85% to 94%); shareable URLs; WFD lake polygons with a lake-size dilution term; recency weighting in the site calibration; production refit on all years; verification against archived forecasts by lead time (below).
Not done yet, in the order I would do them:
services3.arcgis.com/KLNF7YxtENPLYVey/.../Spill_Prod__view/FeatureServer/0,
2,362 overflows, status as text, no token). It carries no licence text, so
ask Dŵr Cymru for reuse terms before building on it. SwimSignal emailed
Dŵr Cymru on 1 October 2026 and is waiting for its terms
(docs/WALES-PLAN-2026-10.md, task W5 onwards). It matters for the
English Wye spots, which have Welsh overflows upstream.uv sync
uv run python -m dipcast.ingest.annual_returns
uv run python -m dipcast.ingest.live
uv run python -m dipcast.ingest.edm_events
uv run python scripts/fetch_rain_archive.py
uv run python -m dipcast.overflows
uv run python scripts/train.py 2025
uv run python scripts/build_site.py
scripts/build_site.py polls the live feeds, forecasts every spot and writes the
site to site/, as the scheduled workflow does. scripts/refresh.py does the
first part alone: it polls the feeds, rebuilds the overflow table and scores logged
forecasts. Mutable
files (live polls, the overflow table, the forecast log, live scores) go to
DIPCAST_STATE if set, else data/processed. All raw pulls are cached under
data/cache/ so re-running the ingestion is cheap. Requests to
environment.data.gov.uk carry the User-Agent dipcast/<version> (+https://github.com/ethanbuckley/swimsignal); a fork should set its own with
DIPCAST_USER_AGENT. scripts/verify_leads.py 2025 reproduces the lead-time table; scripts/validate_ecoli.py then
scripts/validate_ecoli_combined.py reproduce the E. coli tables.
Every forecast is logged (coordinates, time, values; nothing about the user)
and scored once its days have passed, using the accumulated live polls. The
result is public at /verification, alongside the offline tests. /terms and
/privacy hold the plain-English terms and privacy notice.
spots.csv lists the spots: the 38 Environment Agency designated inland
bathing waters and about 50 well-known river and lake spots. A river spot
names its river in the river column (see Spot placement). Add one by pull
request, or ask for one with the "Request a spot" issue template; it appears
in the next run. Inclusion is not a statement that a spot is safe.
A request made with that template gets one automatic comment
(.github/workflows/spot-request.yml runs scripts/spot_request.py). The
script reads the form's Location as latitude and longitude (decimal, degrees
minutes and seconds, or a Google Maps address that holds them) or an OS grid
reference, places the point with locate_pin as the build places a
spots.csv row (a river named in the spot's name is the hint), counts the
monitored overflows within 60 km upstream, runs build_site.placement_check
and proposes a spots.csv row. A location it cannot read gets a request for
coordinates, and editing the issue runs it again and updates the same
comment. A what3words address or a place name is looked up only when the
repository secret W3W_API_KEY (what3words) or OS_API_KEY (OS Names API)
is set. The workflow answers only issues with the spot-request label, which
the form adds only if that label exists in the repository. It can also be run
from the Actions tab with the form's fields; the answer then goes to the run
summary.
spots-osm.csv adds 16 spots taken from OpenStreetMap, with the same columns
plus osm_id, ids starting osm-, and source "openstreetmap". It is a
separate file because the ODbL would cover any file that mixed OpenStreetMap
rows with ours (LICENSE-DATA.md). scripts/osm_spot_candidates.py asks the
Overpass API for England's bathing places, swimming areas, sport=swimming
and swim-like names, drops pools, signs, the sea and duplicates within 150 m,
places each on the network and writes data/raw/osm_swim_candidates.csv
(115 candidates on 3 Oct 2026). A person then reads every row and copies the
natural river and lake spots the public can reach into spots-osm.csv, with
river typed for each river spot. The build reads both files, runs
placement_check on both, and credits OpenStreetMap on the terms page, in
credits.spot_locations and on each such spot's page.
Any other point (October 2026). A click on the map away from a listed spot
gets a forecast too, worked out in the browser (src/dipcast/site/anypoint.js)
from files the build writes under data/anypoint/
(scripts/build_any_point.py, whose docstring lists them): every river
link's upstream overflows, in a form that gives the API's distances and
dilutions for a click anywhere along the link; a packed index of the links,
by 0.25° square, for snapping a click; the WFD lakes with their inlets; and
every overflow's spill probability for each day. The page then does what the
API does with gauge=False: the same placement (lake polygon, nearest link
within 1.5 km, a side channel traced as the main river), the same transport,
calibration and combination, and the same card, marked "Unlisted point: not
hand-checked", without the E. coli estimate (it needs rain at the spot
itself, which only listed spots get). The overflow data covers England only,
so a click in Wales, Scotland or anywhere on the island of Ireland, or on
their estuaries, gets "SwimSignal has overflow data for England only" before
any river is looked for (data/raw/outside_england.json, made by
scripts/make_outside_england.py from the Office for National Statistics'
country boundaries, OGL v3, and Tailte Éireann's provinces of the Republic,
CC BY 4.0). A click in Northern Ireland names DAERA's bathing water dashboard,
and one in the Republic names the Environmental Protection Agency's
beaches.ie. Until 4 Oct 2026 a click on the Taff in Cardiff or the Tweed at
Kelso, in a square with files, found no overflow upstream and read "No sewage
risk from monitored overflows", and a click in Belfast or Dublin, with no
square near, read "No river or lake near this point has a monitored storm
overflow within 60 km upstream". A click in England with no square near has no
monitored overflow upstream of any water there, and says so. The tracing runs once per
network release and is cached in the state directory (anypoint_links.pkl);
only links downstream of an overflow that moved are traced again. Rain for
all overflows is about 1,600 Open-Meteo cells against the spots' 300, and on
3 Oct 2026 Open-Meteo refused this build after 600 locations in one minute,
so each build fetches at most 300 more cells, oldest first, and uses cached
ones up to 24 hours old; an overflow with neither counts as having no rain
data, and the card says when its rain is older than the issue time.
Each spot has its own page, spot/<id>/, written by build_site.write_pages:
the same map page with the spot's name, a one-line description and absolute
share-preview tags in its head (today's level is left out, because messaging
apps keep a preview for days), plus sitemap.xml. Links shared from the site
use these addresses, and old ?spot= links are rewritten to them. Every page
sets a relative <base> at the site root (./ at home, ../../ on a spot's
page). The page moves between the list and spots with pushState, and a
<base> is fixed as the page loads, so its links do not follow the address
into spot/<id>/; being relative, the pages work at any address.
Share links, previews and the sitemap need the absolute one: the repository
variable DIPCAST_SITE_URL (set it when adding a custom domain) or, unset, the
repository's GitHub Pages address. icons/og.png is the preview image
(uv run --with pillow python scripts/make_share_image.py redraws it after a
name change). cards/ holds pictures of the forecast to post to Instagram and
Facebook groups, a weekend card per area and a storm card while overflows upstream
of spots are discharging, each with a caption (src/dipcast/cards.py; the page
cards/index.html is unlisted; uv run python scripts/make_cards.py --out <dir>
makes them from the live site's data). sw.js keeps the page and the latest forecast on the device, so
the home-screen app opens without signal and says how old the forecast is. A
saved spot is kept in the browser's local storage (and by the alert service, if
alerts are turned on), and saved/ lists the
saved spots with their five days; saved/#spots=a,b offers a list someone
shared (after the #, so it never reaches a server). An iPhone's Home Screen
app has storage of its own: spots saved in Safari do not appear in it (checked
in the iOS 27 simulator, 29 Sep 2026), and the Saved page says so. The Feedback page opens the reader's email app with a message to
[email protected] (the site has no server, so nothing is sent by the
page itself) and asks whether the forecast changed what the person did.
The level on the map and in the list is the worst of four things, worked
out by src/dipcast/site/levels.js (risk()), which the page loads and the
build runs in Node for the alerts, so the two cannot disagree: the spill forecast's level; on
rivers with overflows upstream, the E. coli column in bands of under 10%, 25%,
50% and over (the minimum inland standard lets about one sample in ten be over
900), in the May-September season only (out of season the column is shown, marked
†, but not counted: see Known limits); the Environment Agency's rating, where "poor" (advice against bathing applies
all season; the local authority that controls the water issues it, not the EA)
makes every day at least high; and the sampler's algae
check if under two weeks old. On 29 Sep 2026 the spill forecast alone had all
nine Thames spots on "low", and all 13 poor-rated bathing waters on "low" for
that day (7 of them for the next day too). The ratings are in
data/raw/bathing_water_classifications.json, written by
uv run python scripts/fetch_classifications.py: run it after each year's
classifications are published (the 2026 ones are due in December) and commit
the file. It is a file rather than a fetch in the build because the EA's
bathing-water service has refused GitHub's runners since 28 Sep 2026 (seen on
its sample endpoint; the ratings come from the same service). The EA's short-term advice
against bathing after an incident is not included, for the same reason and
because the service sends no CORS header, so a browser page elsewhere cannot
read it either: on 29 Sep 2026 it covered Ham and Kingston and Frensham Great
Pond (algae). Each bathing water's page links to the EA's page instead, and
its "Today's EA advice" fold (src/dipcast/site/eatoday.js, 4 Oct 2026) loads the
EA's own embeddable panel for the site in the reader's browser when opened: a
sandboxed frame the page cannot read, so it never sets the level.
Where there is nothing to forecast (3 Oct 2026). A spot with no monitored overflow within 60 km upstream, and an isolated lake, which the river network links to no monitored overflow, both get a plain level of their own, "No monitored overflow found upstream" (teal; since 12 Oct 2026). Before, they read "No sewage risk from monitored overflows", which a red-team review read as "no risk", and "No monitored overflow drains into this lake" (grey; "No river connection: overflows cannot reach this lake" until 11 Oct 2026, which was false at a tarn with becks the network does not draw). Under either, the page says "Other risks apply: algae, wildlife, runoff and bathers. Check the signs at the water." and gives the rain in the 48 hours to midday today as "12 mm of rain in the last two days". Neither shows an E. coli estimate, since the model was fitted on sites with overflows upstream (before this, the day-by-day table showed the build's figure on a river with nothing upstream: 35% today for the Duddon at Birks Bridge on 3 Oct). A rating of sufficient or poor, or a recent algae check, still sets the level where it raises it, but an excellent or good rating no longer makes such a spot "low". In the 3 Oct 14:48 local build that was 17 and 2 of the 89 spots; 7 of them (Colwick, the three Hampstead ponds, the Serpentine, Henleaze and Cotswold Country Park) had read "low" by their rating, so they now leave the "Low risk only" count and are not offered as "Lower risk nearby".
One day. A spot's five days are buttons. Picking one shows, in place of the
summary of today and tomorrow, that day's level and what set it: the spills
(how many overflows are expected to spill, the exposure index, and the three
overflows most likely to reach the spot that day, from each overflow's spill
chance for every day, p_spill_days), the water quality, the rain, and how good
a forecast that far ahead has been. That last is lead_skill in spots.json,
each day's Brier skill as a share of the same-day forecast's, from
verification_leads_2025.csv (spills, lead-calibrated and cross-fitted) and
ecoli_model_eval.json (water quality on rivers), held from rising with the
days: 1.0, 0.84, 0.79, 0.70 and 0.59 for spills and 1.0, 0.75, 0.69, 0.55 and
0.55 for water quality on 29 Sep 2026. The open day is in the address
(spot/<id>/#day=2026-10-01), so a shared link opens it, and a day on a Saved
card opens the spot on that day. The map has a day picker too, which colours
the spots, and the "Low risk only" switch, by one day. On a phone a spot's page
starts with a small map of where it is; a tap opens the full map there.
Beside the forecast (1 Oct 2026). Each spot's page also carries, as
observations and context rather than inputs to the level: the Environment
Agency's nearest level gauge, preferring one on the spot's own river
(flows.nearest_level_station with a river-name preference; without it
Burnsall on the Wharfe got Hebden Beck, a tributary 3 km away, over Netherside
Hall on the Wharfe 6 km up), with the latest level, the gauge's usual range, a
word for where the level sits and a link to the EA's page for it; on 1 Oct 62
of 89 spots had one, 31 on the same river. The spot's river is the river
column, so Symonds Yat now asks for a gauge on the River Wye, not on
"Afon Gwy". The EA's "latest" reading can be weeks old: on 2 Oct Salisbury's
was 708 h old and Temple Sowerby's 77 h. A reading more than 24 h old is not
shown as the level now. It keeps the station and the time, moves the value to
last_level_m, sets stale: true with age_hours, sets level_m and
index to null and label to "unknown" (flows.reading_fields). The API's
own forecast publishes the reading the same way and does not use it to scale
travel speed. The EA's API is OGL and asks for the
line "this uses Environment Agency flood and river level data from the
real-time data API (Beta)", which the footer, the terms and the data credits
carry. Open-Meteo's daily high, sunrise and sunset (three daily variables over
five days weigh one call per spot per build). Both are fetched by
build_site.attach_river_levels and attach_weather after the forecasts, in
the refresh step only, and a failure leaves a spot without them. From the
evening of 3 Oct 2026 the build asks for every gauge's latest reading in one request
(/data/readings?latest¶meter=level, about 4,100 measures, 1.3 MB) and
keeps each spot's gauge and usual range for a week in
state/cache/ea_level_stations.json, using an older pick for up to 30 days
while the stations lookup fails (flows.level_lookup). Before, it asked three
times a spot, about 300 requests a build; on 3 Oct 8 to 79 of them a build met
HTTP 403 or a timeout, and the build that published at 21:45 had a level for
63 spots where the bulk request gave 104 of 105. A gauge's stage reading is
taken before any other level it publishes, and metres above its datum (m,
mASD) before mAOD, because the usual range is for the stage above its datum.
The list and
the Saved page say which day this week has the most spots at low (bestDay in
levels.js). A spot's page can draw its forecast into a picture for a swim
group's chat (a canvas on the device; nothing is uploaded until the share
sheet) and log "I swam here today", kept in the browser with the day's level
and listed on the Saved page, where each entry links to the feedback form with
the spot and the day filled in.
Too high to swim (3 Oct 2026). A high river and a flood are a different
hazard from pollution, so they never change the level; a spot's page says them
in one line under the headline, kept in view on every day's view, ending "A
separate hazard, not part of the pollution level." "River high: the gauge at
Addingham is above its usual range" when the latest reading at a gauge on the
spot's own river is above the top of its usual range (index over 1.0). "River
rising fast: the gauge at Addingham rose 0.40 m between 08:45 and 14:30" when
the gauge's readings in the six hours before the build (the measure's last 24,
_limit=24, 15 minutes apart) rose by more than a fifth of the usual range.
"Flood alert in force here (Environment Agency): Upper River Wharfe" for the
most severe flood alert or warning in force whose area covers the spot, linked
to the area's page on check-for-flooding.service.gov.uk. The alerts come from
/id/floods?lat&long&dist=10, whose dist is not measured to the area's
outline: on 26 Oct 2026 it gave the Wharfe at Burnsall "Lower River Ure"
(122WAF940, 16.7 km away) and Dartington on the Dart a South Devon coastal alert
(113WACT1B, 8.8 km). So the build reads each in-force area's outline, the EA's
own map of where the alert applies, and
flows.split_flood_alerts names an alert only when the outline covers the
spot's point on its water (location.snapped, else the pin) or, for a lake,
reaches the lake's own outline (data/processed/lakes.parquet; the pin for a
lake with none). Three of the four Wharfe pins sit just outside "Upper River
Wharfe" while their points on the river are inside, which is why the point on
the water is used. An alert outside that names the spot's own water (by its
floodArea.riverOrSea or, by the whole name, its title), or one with no
outline that is not plainly for other water, is counted in
flood_alerts_unmatched, and the page says "A flood alert or warning is in
force that could not be matched to this spot" with a link to the EA's flood map
(never "nearby": an unmatched area can be 11.6 km away, as at Aysgarth).
An outline is asked for only at https://environment.data.gov.uk/flood-monitoring/id/floodAreas/<id>/polygon,
for an id that is strictly an EA flood-area id (three digits, then letters and
digits), with redirects not followed. It is kept, simplified to 10 m, with its
fetch time, in config.CACHE/flood_areas/<id>.geojson, and every attempt's
time, success or failure, goes in _attempts.json there. Each area is asked
for at most once a week while the build's state cache carries over (best
effort). Each attempt's time is written to the record just before its request,
so a failed request, a later failed step or a cancelled run still leaves it for
the workflow's always() cache save; cache eviction (7 days unused or the 10 GB
repository limit), a lost runner or a failed cache save (a re-run of the same
run_id included) resets it for the areas involved. Whatever happens, a build
asks for at most 8 outlines, and an area with no outline (at most 30 days old)
is never named. A kept outline under
a week old is used as it is. An
outline not asked for, or whose request failed, is used only while it is at
most 30 days old, else the area has no outline and its alerts are never named.
That holds for a first fetch too: one failed request leaves an area without an
outline, its alerts counted or said neutrally, for up to 7 days. At most 8
areas are asked for in one build (one wave of the 8 workers), those with no
usable outline first, then the oldest, then by id; the rest are not asked and
not recorded, and are never named without an outline at most 30 days old. The
build warns "flood outlines: N areas unavailable ...; M areas not asked this
build". Cost: per build at most 8 GETs of up to ~0.9 MB raw each, only on
days with alerts in force (otherwise the national list shortcut asks nothing
per spot), so even with a cold cache at most 384 GETs a day at 30-minute
builds, about 346 MB a day at the largest outline seen; normally one GET per
area per week; the cache at most 500 outlines, tens of MB. Pruning touches only files and
directly in that directory: it keeps the areas in force in the build and
entries fetched within 30 days, removes older ones, keeps at most 500 (over 3.5
times the 137 items the national list held on 26 Oct 2026), and drops attempt
times over 30 days old for areas not in force.
With no alert in force, the decision record says which search found none: "The
Environment Agency listed no flood alerts or warnings in force anywhere in
England" (the national list, when nothing is in force) or "The Environment
Agency's search within 10 km returned no flood alerts or warnings in force"
(the spot's own search; ). One for other water,
or tidal () for an inland spot, is counted in and
never named. Severity 4,
"Warning no longer in force", is left out: on 3 Oct 2026 it was the only item
the service held for England. The build asks spot by spot only when the
national list holds an alert in force, so on most days the floods cost one
request. That request goes first, before the river levels: asked after them it
was refused in 6 of the 8 builds from 16:26 to 21:42 on 3 Oct 2026, leaving
every spot unchecked. When the EA does not answer, the page says "Flood alerts
not checked" and links to the flood-warning service rather than saying
nothing, which would read as none in force, and the build log warns. A gauge on another watercourse, or a lake's,
gives no river word. The words are and ,
fetched by after the levels, in the refresh step
only; the comparison's "Local warnings" row names the flood alert too. The page
( in ) says nothing from a reading over 24 h old, a
rise whose last reading is over 6 h old, or a build over 24 h old, so a stale
reading never says "River high". The flood API is the same OGL service the
river levels come from, already credited.
Water temperature (3 Oct 2026). build_site.attach_water_temperature
gives a river spot the latest reading of the nearest Environment Agency
water-quality sensor within 15 km (straight line) whose river has the spot's
river's name (same_river), that the network joins to the spot, on the same
side of the tidal limit, read in the last 24 h (ingest/water_temperature.py).
Two Hydrology API requests per build: the temperature readings since
yesterday (about 1 MB) and the active temperature stations (about 0.5 MB). The
sondes have no riverName; the river and the place come from the label
("STOUR_BURES MILL_E_201704"). The API's times have no zone and are UTC (its
Evesham level series matched the flood-monitoring API's Z times on 3 Oct).
The page shows "Measured at Bures Mill on the Stour, 10.9 km downstream, 1 h
ago" under the figure, with the age counted when the page is read; a spot with
no sensor shows nothing, never an estimate, and lakes get none. In a local
build on 3 Oct, 99 sensors had read in the last day and 2 of the 58 river spots
got one (Friars Meadow and Dedham, both on the Stour); the six Thames sondes
within 15 km of Ham and Kingston are all on tidal links and are passed over.
A failed request leaves every spot without and does not stop the build; the
build summary's water_temperature is the count.
Plan a swim (3 Oct 2026). plan/ asks for a day, where you start and how
far you will go (5 to 50 miles, or any distance), and the kind of water, then
lists the spots within reach in three groups: nothing flagged that day (low
risk, or a plain level), moderate risk or higher, and no level. Each group is
nearest first, the second by level first; each row gives its distance, its
headline for the day and a line on why (planWhy in index.html: the spills
expected that day, the E. coli estimate in season, the EA rating, access not
confirmed, and the river high on a plan for today). Distances are in a
straight line. The rules are plan.js (tests/site_plan.test.cjs). The plan
is in the address after the #, so a link opens it; a place carries its own
position (from=Kendal&at=54.33,-2.75), and your own location is written
from=here, never as coordinates, and never moves the map. The places are
data/raw/places.json, from OS Open Names (OGL): 26,211 cities, towns,
districts, villages and other settlements in England and Wales, 771 kB (239 kB
compressed), copied to site/data/places.json and fetched the first time a
place is typed. scripts/make_places.py rebuilds it from OS's 103 MB download.
Postcodes are not in it. If more of the spots in a plan are low on another of
the five days (two more and a quarter more), the count line offers that day.
Alerts (push/, set up by hand: push/README.md). A Cloudflare Worker
keeps, for each browser that turns alerts on from the Saved page, its push
address and the ids of its saved spots, and nothing else. Each build writes
data/alerts.json, every spot's level and headline by levels.js
(scripts/alerts.js, which needs Node; the runners have it). Every 2 minutes
the Worker compares it with the previous one and queues a notification for any
saved spot that has just turned high or very high, at most once a spot in 20
hours, and sends them 15 a run (about 450 an hour on Cloudflare's free plan). The page shows the switch only when the repository variables
DIPCAST_PUSH_URL and DIPCAST_VAPID_PUBLIC_KEY are set, and the privacy
notice gains its alerts section only then. On an iPhone, alerts work only in
the Home Screen app. The same Worker can send the alerts by email, from the
same list of risen spots, after a double opt-in and with an unsubscribe link in
every email; it is off until set up (push/README.md, "Email alerts"), and
the page offers it only when DIPCAST_EMAIL_URL is set.
Reviews (reviews/, deployed 3 October 2026; moderation and setup: reviews/README.md). A swimmer can
say whether they would swim at a spot again, when they swam, what it was like,
and add up to three photos, which the page shrinks and strips of their camera
data on the phone. A second Cloudflare Worker holds each review until the
operator publishes it on its /moderate page; each build then copies the
published reviews and photos into site/reviews/ (src/dipcast/reviews.py),
so reading them never contacts the Worker. A spot's page shows the share who
would swim there again once three have reviewed, the reviews, and the form
(src/dipcast/site/reviews.js). It is off until the repository variable
DIPCAST_REVIEWS_URL is set; the privacy notice and the terms gain their
reviews sections only then.
How it looks is set out in docs/DESIGN.md: two typefaces served from the
site itself (Source Serif 4 for headings, Source Sans 3 for the rest, in
src/dipcast/pages/fonts/ under the SIL Open Font License, so no third
party sees a request), one corner radius, hairline borders, the four level
colours as the only strong colours. The prose pages share
src/dipcast/pages/page.css; the app page repeats its tokens and header
inline so that it paints before any stylesheet arrives and works offline on its
own, and a token changed in one must be changed in the other. The prose pages'
favicon and Home Screen icon are the site's own, in src/dipcast/site/icons/.
Data files and the embed (3 Oct 2026). data.html lists every file under data/, what it
holds, its fields and its licence. Its words are src/dipcast/pages/data.html;
build_site.write_data_page fills in each file's size and adds a row, with a build warning, for any
file in data/ the page does not describe, so a file added to the build belongs on that page too.
embed.html?spot=<id> is one spot's card (headline, five days, caveat, issue time, a link back and
the credits) for a club's or a council's site to show in an iframe. embed.js draws it with
levels.js, so it says what the spot's page says. The About page gives the snippet, and the terms
allow it with the credits intact.
.github/workflows/site.yml is scheduled every 30 minutes and also runs on
every push. GitHub starts scheduled runs when it can: the 113 builds of 13-28
Sep 2026 were a median 2.9 h apart and at most 8.1 h, so no step depends on a
run starting at a particular time, and the map's "Stale" banner waits until the
forecast is 8 hours old (at the earlier 2 hours it would have shown 48% of the
time; at 8 hours, never in that period). The job restores the mutable state (live polls, forecast log, rainfall cache) from
the Actions cache, or from the rolling state release if the cache is cold;
downloads the river network from the data-v1 release (113 MB, too big for
git); runs scripts/build_site.py, which polls the nine live feeds, rebuilds
the overflow table, forecasts every spot with forecast_point, scores logged
forecasts against the accumulated polls, and writes site/; saves the state
back to the cache and, when the release copy is over 12 hours old, to the
release (GitHub runs the schedule only a few times a day, at irregular times);
and deploys site/ to GitHub Pages, or to Cloudflare Pages once the secret
CLOUDFLARE_API_TOKEN is set (docs/HOSTING-MOVE-2026-10.md, the move for
paid pilots). Nothing is committed by the job except
HEARTBEAT.md, so the repository does not grow. The first scheduled run after
that file is 25 days old rewrites it, because GitHub disables the schedule of
a public repository after 60 days without activity.
The site is static and costs nothing to host. A FastAPI server for forecasts at
arbitrary points ran on Fly.io until its trial ended on 29 Sep 2026. It was removed
on 20 Oct 2026 (review task F7): the map's click-anywhere forecast is built into the
static site (anypoint.js), and nothing else used the server.
Checked on 29 Sep 2026 against the licences, the legislation and the providers' own pages, in a solicitor-style review by an AI assistant: not legal advice, and no substitute for a solicitor before any of these happens.
models=) and
re-check the spill and E. coli models against its rainfall, or get this
reading confirmed.docs/HOSTING-MOVE-2026-10.md.DIPCAST_OPEN_METEO_KEY to the plan's key and every Open-Meteo request goes to
the customer- host (customer-api.open-meteo.com and so on) with apikey=;
unset, the build uses the free hosts as before. The key is kept out of logs,
errors, cache names and pages (tests/test_open_meteo_key.py). The day's
ration stays 9,000 calls until DIPCAST_OPEN_METEO_DAILY_CALLS is raised.| Open-Meteo |
| Hourly rainfall: ERA5-Land archive for training, model forecast for prediction, and the Met Office's MOGREPS-UK ensemble (three members) for the listed spots' band on days 3 to 5 |
| CC BY 4.0; the forecast and the ensemble for England are Met Office data, CC BY-SA 4.0 |
| OS Open Rivers | 193,040 directed watercourse links incl. lake traversals, BNG | OGL v3 |
| EA WFD Lake Water Bodies Cycle 3 | 564 lake polygons (lakes over 50 ha, 5 ha in protected areas), names and areas | OGL v3 |
| HydroLAKES v1.0 (Messager et al. 2016) | Residence times of 282 of those lakes (data/raw/lake_residence.csv, scripts/make_lake_residence.py) | CC BY 4.0 |
| EA Hydrology API, daily mean flows | Spill-day flow ratios of those lakes, once (data/raw/lake_residence_spill.csv, scripts/make_lake_spill_ratio.py) | OGL v3 |
| EA flood-monitoring API | Near-real-time river levels and typical ranges; the API version only | OGL v3 |
| EA Hydrology API | Water temperature from water-quality sensors that reported in the last day, beside the forecast | OGL v3 |
| EA Catchment Data Explorer | Reasons for not achieving good status per water body, and the water bodies' river lines and catchments: the water body each spot is in, and where agriculture and rural land management is a confirmed or probable reason, a line on its page (data/raw/rnags.json, scripts/make_rnags.py, off CI) | OGL v3 |
| Water companies' own pages on overflow upgrades | One row per overflow a company says it has upgraded or is upgrading, with the page, a short quote and the day it was read (data/raw/overflow_upgrades.csv; six Yorkshire Water overflows above Ilkley). A note beside the overflow on a spot's page; the forecast never reads it (docs/UPGRADED-OVERFLOWS-2026-10.md, option C) | Short quotations, each linked to its source; nothing republished |
| EA Water Quality Archive (Water Quality Explorer) | E. coli results and the sampler's visual algae check at the 38 inland bathing waters | OGL v3 |
| OpenStreetMap (Overpass API) | 16 river and lake swim spots chosen one by one from 115 candidates (spots-osm.csv) | ODbL 1.0, © OpenStreetMap contributors |
dipcast.network.fixesdata/raw/network_fixes.csvscripts/check_network.pydata/raw/overflow_joins.csv); Rochester Road CSO now joins the Rivelin
241 m below its plunge pool, not 0.7 km above it. Each build lists, in
build.warnings, any counted outfall snapped 200 m or more onto a link not
named like its receiving water and nearer the spot along the river than that
distance plus 300 m: where its stream meets the river is not known, so it may
be below the spot.dipcast/ids.py resolves 93% of rows (100% of UU 2023, 92% of UU 2021-22) via the Hub lookup, then site name and grid reference.docs/NINE-COMPANY-MODEL-TEST-2026-10.md).
The spill model was trained on United Utilities only; the live verification page
scores every company's live-feed overflows separately ("By water company"),
which is the running check that it transfers. (Until 15 Sep 2026 the live
scorer scored nothing: DuckDB returned its DATE columns as timestamps and the
join to observed spill days silently matched no rows. Fixed, with a test.)<id>.geojson_attempts.jsonflood_alerts_searchisTidalflood_alerts_otherflows.flow_stateflows.flood_alertsbuild_site.attach_flow_stateflowFactsexperience.jspush/README.md, "Email alerts"); the privacy section is added
only when they are on, and needs the operator's approval first.DIPCAST_KOFI_URL is a Ko-fi page (https://ko-fi.com/<name>). Then the build
writes support.html, puts "Support SwimSignal" in every page's foot and a short
section on About, and adds "If you leave a tip" to the privacy notice (Ko-fi's
privacy policy: a tip passes the supporter's name or username and email address
to the creator, who is then a controller). It is a plain link: no Ko-fi script
or widget loads. Before setting it:
DIPCAST_TILES names a provider in TILE_PROVIDERS (scripts/build_site.py)
and DIPCAST_TILE_KEY holds its key; a bad name or key leaves OpenStreetMap on
with a warning. The swap changes the app's tile layer and credit, the privacy
notice's list of who sees your IP address, and the terms' map credit. Lock the
key to swimsignal.co.uk at the provider: it is public in the page.
carto-voyager, carto-positron: CARTO, free to 5 million tile requests a
month for non-commercial use and 1 million for commercial, where commercial
includes a site "that generates revenue". Without a key every tile shows
"API key required". CARTO may suspend free access "at any time".thunderforest-atlas, thunderforest-outdoors: Thunderforest's Hobby
plan, 150,000 tiles a month; commercial use "permitted and encouraged".--tile-filter in index.html). Under CARTO's colours it left the sea near
white, so each CARTO preset carries its own filter (CARTO_FILTERS, measured
on CARTO's tiles on 5 Oct 2026), which the build puts in its place: grey land,
water #bccfd8, as on OpenStreetMap. Thunderforest keeps OpenStreetMap's filter,
unchecked; check it at 320, 375 and 1440 px before switching to it.carto-voyager (Ethan, over Positron, whose labels are
faint once grey).COUNTER_TOKEN_ENV in scripts/build_site.py).