An independent assurance audit of OmniCloudMask
(v4 weights, MIT), an open cloud and cloud-shadow segmentation model for Sentinel-2 and
other 10-50 m imagery, run the way a client would commission it: before you automate on this
mask, how much cloud leaks into the pixels you accept, how much usable ground do you throw
away, and where does it fail?
Every number below is computed by this repository from public data; nothing is typed in by
hand. The client-style report is in report/, and the
threshold explorer lets you drag the operating point.
Why this matters
Almost every optical imagery pipeline starts by deciding which pixels and which scenes to
use. Cloud and shadow that slip through become false change, wrong indices and bad training
data; clear ground that gets masked is lost revisit. The decision is usually made by a mask
nobody has measured on the buyer's own terms.
What was tested
- Model: OmniCloudMask v4 (two U-Nets, red/green/NIR input), fp32.
- Data: CloudSEN12+ (CC0), expert-labeled test split: 960 scenes at 192
sites, 509 x 509 px at 10 m, both L1C and L2A. The validation split is used only to choose
operating points and the buffer size.
- Training-overlap screen: OmniCloudMask's published training set includes CloudSEN12
scribble and 2k patches from all splits, so the evaluation scenes were checked against its
training records (100,528 rows, pinned revision): 0 shared scene
ids (including 2,754 test/validation-tagged training rows) and no
spatial overlap with any CloudSEN12 training patch. 16 test acquisitions
(tile and date) also supplied other, non-overlapping CloudSEN12 patches to training
(56 training rows); same-date imagery nearby is a residual dependence.
No acquisition is shared with the non-CloudSEN12 training sources (0
matches), and 10 evaluation images whose footprint may overlap a custom
training scene (conservative disc around its named centre) were removed. Kappaset names only its ~110 km tile; removing the 50
test images on those tiles gives leak 6.2% (headline 6.1%). This is a
screen with stated limits, not proof that no overlap remains.
- Baselines (as shipped with CloudSEN12+): Sentinel-2 QA60, Sen2Cor SCL, s2cloudless
(upstream default post-processing), Cloud Score+, UnetMobV2 and SEnSeI v2. UnetMobV2 was
trained on CloudSEN12 itself, so it is in-domain like OmniCloudMask.
- Metrics, framed as the decision: leak = share of accepted (called clear) pixels that
are cloud or shadow; discard = share of truly clear pixels thrown away; miss = share of
cloud/shadow pixels accepted. Every method is scored on the same pixels; abstentions count as
discarded. Scene policy: keep a 5.1 km patch if its estimated cloud+shadow cover is at most 10%.
- Statistics: 95% CIs by bootstrap over sites (5 dates per site are not independent);
method comparisons are paired.
Results
Headline (test, L1C): leak 6.1% [5.3, 7.0], discard
4.6% [4.0, 5.3], balanced accuracy 0.942 [0.936, 0.947].
| Mask | Leak | Discard | Balanced acc. |
|---|
| OmniCloudMask v4 | 6.1% | 4.6% | 0.942 |
| SEnSeI v2 | 6.2% | 6.2% | 0.934 |
| UnetMobV2 v2 (in-domain) | 7.1% | 5.3% | 0.933 |
| Cloud Score+ (cs >= 0.60) | 8.5% | 12.7% | 0.891 |
| s2cloudless (upstream defaults) | 23.2% | 7.0% | 0.808 |
| Sen2Cor SCL | 28.5% | 10.9% | 0.747 |
| QA60 | 32.5% | 15.0% | 0.696 |
Paired against OmniCloudMask: SEnSeI v2 leak +0.1 pp
[-0.4, 0.6], discard +1.6 pp
[0.9, 2.2]; Sen2Cor SCL leak +22.4 pp
[21.0, 23.8].
An interactive version (drag the threshold, watch leak and discard move) is generated at
site/explorer/index.html, a single self-contained page for the website.
Operating points (chosen on validation, reported on test). For a 2% leak target,
OmniCloudMask: accept if rint(255 x P(clear)) ≥ 218 in float32, i.e. P(clear) ≥ about 0.85294, giving leak 2.0% [1.4, 2.6]
and discard 13.6% [12.4, 14.9]; Cloud Score+ discards
59% at the same target. Thresholding s2cloudless's raw probability (no
averaging or dilation) never gets leak below 7.2%; it has no shadow class.
Where it fails (real data):
Stress test (simulated): change versus the clean run, paired over the same scenes;
n.s. = 95% CI includes zero. Degradations are simple, controlled simulations; they show
sensitivity to each change, not the behaviour of a particular sensor or pipeline.
This is the beginning of the README. Read the rest at the original. Read the full README ↗
