contested-nav

Is the filter right about how wrong it is?

In 60 seconds

Problem. A navigation filter has to be right about its own position and also right about how sure it is. A drone that has lost GNSS and is steered by a filter that is wrongly confident will trust a position it should not.

Decision. Should camera-based motion tracking be switched on when GNSS is denied? With the single-anchor model it does not reliably make the position error smaller, and the filter claims to know its position to within centimetres while the error is metres (the table below is one run with one start draw). It is confidently wrong, so it ships off by default (ADR-0003).

Evidence. In the sweeps the overconfidence held for every one of 10 seeds and every one of 40 outage runs (measured once on one machine, not checked by CI, at 946d661; see what the sweeps show). The tables on this page are generated by navkit run --markdown, and CI fails if a cell disagrees with the code. They are synthetic. The filter has also been run on recorded IMU data (EuRoC and TUM VI) with GNSS simulated from the ground truth; see Status and limits. None of it is a field test.

Next step. A stochastic clone of the previous pose is built and stays opt-in (ADR-0017). It passes the S1 gate of ADR-0030 on the development seeds (25 to 49), where the visual errors are independent. It has not been evaluated on held-out seeds. Against the measured errors of a real two-view front end it needs the assumed noise inflated (ADR-0023). That evidence is on recorded-data windows, a different fixture from the synthetic one, and ADR-0033 does not change it. The gate for turning vision on is no longer the old single-run bar. ADR-0028 and ADR-0030 govern it, and the B1 status is in CONSTRAINTS.md. The roadmap calls the problem B1, closed as characterised, not solved (ADR-0036).

A 21-state error-state Kalman filter and an evaluation harness for GNSS-denied navigation. It measures not only how accurate the trajectory is, but whether the uncertainty the filter reports is honest. With GNSS denied and visual odometry on (the single-anchor model), the filter claims centimetres of uncertainty while its error is metres. Every ESKF case here starts at the truth plus an error drawn from the filter's own declared prior (ADR-0027), so the table below is one run with one start draw (seed 0). It is descriptive:

Configuration	ATE RMSE, full run	Claimed 1σ, full run	Mean NEES, full run (expected 3)	2σ-per-axis coverage, full run (expected 99.3%)	Mean NEES, outage window (expected 3)	2σ-per-axis coverage, outage window (expected 99.3%)
GNSS denied, vision off	6.530 m	0.567 m	3.2	100.0%	4.3	100.0%
GNSS denied, vision on	17.568 m	0.156 m	14018.6	19.2%	2783.3	5.0%

Calibration failed, and accuracy did not improve. The filter is confidently wrong, so visual fusion ships off by default. In this single run the aided case is also less accurate than the case without vision. Over 10 seeds, each with its own start draw, it was less accurate in 9 (measured once on one machine, not checked by CI, at 946d661), so whether vision helps accuracy depends on the start draw, and here it mostly does not. The overconfidence held in every run of every sweep (see what the sweeps show).

A stochastic clone of the previous pose (opt-in, vision_model: clone, ADR-0017) is the candidate fix. The pre-registered held-out gate (ADR-0028) was run once and failed, and the result was accepted rather than tuned away. A single run cannot establish calibration, so the table lists what was measured and its status. It is all synthetic, the default is unchanged, and visual fusion still ships off. B1 is closed as characterised, not solved (ADR-0036): work towards a default-on configuration has stopped, and the tools stay in the repository.

What the evidence shows

Configuration	What was measured	Status
Single anchor	overconfident in every run of every sweep; does not recover from a start attitude error inside its own declared prior (the ensemble over seeds, what the sweeps show)	fails; kept as the control
Stochastic clone, independent visual errors	passes the S1 gate of ADR-0030 on the development seeds (25 to 49) (the candidate gate)	not evaluated on held-out seeds
Clone with the measured-front-end noise preset	conservative; failed the held-out gate of ADR-0028 on condition 2, on the fixture before ADR-0033 (the decision run)	recorded, accepted as final (B1 in CONSTRAINTS.md)
Stochastic clone, visual errors correlated over time	fails the S2 gate of ADR-0030 on the development seeds (the candidate gate)	not solved; work stopped (ADR-0036)

Mostly synthetic evidence. The tables on this page come from a deterministic known-answer fixture: an analytic trajectory with generated sensor streams. The recorded-data pages use a real IMU with simulated GNSS and, in one study, real camera frames. No number here is a field measurement.

How it works

flowchart TB
    IMU["IMU, always on"] --> F
    GNSS["GNSS, can be denied"] --> FDIR["Fault detection: screens each measurement"]
    CAM["Camera motion, opt-in"] -.-> FDIR
    FDIR --> F["Navigation filter: position, and how sure it is"]
    F --> ACC["Accuracy: error against the truth"]
    F --> HON["Honesty: is the truth inside the claimed area?"]
    TRUTH["Known truth: synthetic runs, recorded datasets"] --> ACC
    TRUTH --> HON
    ACC --> GATE{"Gate B1"}
    HON --> GATE
    GATE -- "met" --> ON["Camera fusion on"]
    GATE -- "not met" --> OFF["Camera fusion off by default (ADR-0003)"]

Every run is scored twice: how far the estimate is from the truth, and whether the truth falls inside the area the filter claims. A feature that improves the first and breaks the second stays off. The project page has an animated figure of the same outage; the module-level design is in docs/architecture.md.

Quickstart

Python 3.11 to 3.13. From a fresh clone this took 18.7 s (clone, environment, install and a three-case run), measured once on one machine, not checked by CI:

git clone https://github.com/Telschow/contested-nav.git && cd contested-nav
python -m venv .venv && .venv/bin/pip install -e .
.venv/bin/navkit run --only gnss_only outage_control outage_visual --markdown

make repro regenerates every figure and table from scratch in about 5 minutes (321 s, measured once on one machine, not checked by CI). More in Getting started.

Results

Synthetic 30 s scenario, 5 Hz GNSS, 20 Hz vision, noisy IMU, seed 0, one start draw, so a single sample: each ESKF case starts at the truth plus an error drawn from its own declared prior, so this table is descriptive and not a typical case. ATE uses alignment=none: no fitted transform absorbs an offset, so the start error is part of what the table scores.

Scenario	ATE RMSE (m)	Claimed 1σ (m)	Mean NEES (exp. 3)	Coverage @ 2σ	Verdict
GNSS only (control)	0.512	0.254	4.2	100.0%	mixed: bulk overconfident, tail consistent
Dead reckoning (no aiding)	1.248	n/a	n/a	n/a	no covariance reported
Anchor as measurement noise (defect)	11.497	0.100	13554.9	8.7%	overconfident†
Vision only	26.727	0.158	32045.8	1.5%	overconfident†
GNSS denied 5–20 s, vision off (control)	6.530	0.567	3.2	100.0%	consistent
GNSS denied 5–20 s, vision on	17.568	0.156	14018.6	19.2%	overconfident†
GNSS denied, 30% camera frames dropped	7.434	0.192	1621.3	22.2%	overconfident†
GNSS denied 5–20 s, vision on, stochastic clone	0.730	0.375	2.2	100.0%	consistent
GNSS denied, 30% frames dropped, re-referenced, single anchor	7.609	0.192	1698.4	22.7%	overconfident†
GNSS denied, 30% frames dropped, re-referenced, stochastic clone	0.403	0.306	1.8	100.0%	mixed: bulk underconfident, tail consistent

This table and the two-row table above it are generated (navkit run --markdown and python scripts/check_doc_tables.py --print-configuration-table), and CI fails if any cell disagrees with the benchmark. On this page, the documentation index, Getting started, the Results page and Status and limits, CI also fails on any other measured number in the text unless it is quoted through a metric marker that scripts/metrics.py checks. Timings and figures labelled as one-off measurements are not checked. Interpretation, the four ATE alignments and the figures are on the Results page. A dagger after a verdict means that the effective sample size behind it is truncated, so an upper bound (ADR-0029). consistent means that the run did not reject calibration: one run cannot establish it, and a single-run verdict is descriptive.

What the sweeps show

The seed, outage and scene sweeps below were re-run on the corrected translation noise (ADR-0033), with the code at 946d661. Their outputs are not committed, and a single run is descriptive until the ensemble statistics that will commit them are decided. navkit sweep seeds, navkit sweep outages and navkit sweep scenes regenerate them. A sentence below that was not re-run is labelled as produced under the earlier draw.

• Not a bad seed. Over 10 seeds, each with its own start draw, outage_visual and vision_only were overconfident at every one (measured once on one machine, not checked by CI, at a992aa7).
• Not a long-outage effect. Over 8 outage windows (start 5 s or 10 s, length 5 to 20 s, 5 seeds each) the visual case was overconfident in all 40 runs, including 5 s outages (measured once on one machine, not checked by CI, at 946d661).
• Not one path. Over 8 synthetic scenes (trajectories), each of the five single-anchor rows read overconfident in every scene (measured once on one machine, not checked by CI, at 946d661).
• No accuracy gain from the single anchor. outage_visual had a larger ATE than outage_control in 9 of 10 seeds, and in all 8 outage windows with the seeds averaged. vision_only was less accurate than dead reckoning in 10 of 10 seeds (measured once on one machine, not checked by CI, at 946d661).
• The start attitude draw is what the anchor model fails on (MEASUREMENT). Over 10 seeds the median ATE of outage_visual and of vision_only is several times larger with the attitude drawn than without it, and close to the full-draw value with only the attitude drawn. The attitude prior is 2 degrees per axis, so the anchor model does not recover from a start attitude error inside its own declared prior. The clone's ATE and NEES do not follow the attitude draw. The table is generated from docs/data/start_draw_study.csv and shown on the Results page. Why the anchor model fails is a HYPOTHESIS and was not tested: gravity leakage from the start tilt error plus absolute information that the anchor model invents.
• A control's verdict is one draw. Over 10 seeds the outage_control verdict was consistent in 5, mixed (bulk overconfident, tail consistent) in 2 and mixed (bulk underconfident, tail consistent) in 3. The mean NEES of outage_visual spanned about two orders of magnitude over the seeds and that of vision_only about one, and the verdict of both anchor rows did not change (measured once on one machine, not checked by CI, at 946d661).

Limits

• Not field validated. The benchmark is a synthetic fixture. The recorded-data studies use real IMU and camera data with GNSS simulated from the ground truth, and the data is not vendored.
• Visual fusion under GNSS denial is not trustworthy. The cause is structural (the visual anchor is a pose the filter itself produced) and the established remedies are not implemented here. With the start drawn from the declared prior, the single-anchor model also does not recover from a start attitude error inside that prior (see what the sweeps show). See Concepts.
• FDIR detects implausible updates but mostly cannot say why. A multipath signature exists as a function (ADR-0022). A slow-ramp spoof is followed by the filter with an IMU and GNSS alone (ADR-0019), and a simulated visual source catches it only when it is fast and the visual errors are independent (ADR-0020). Produced under the earlier draw of the synthetic visual translation noise (ADR-0033); not regenerated.
• Not flight-ready: no sensor driver, no live front end, no real-time loop.

All of it, with sources, is in Status and limits and Limitations.

How this was built

The code was written mostly with an AI coding assistant (Claude Code). The maintainer sets the scope and the order of work, and the project's rule is that the maintainer writes and accepts the decision in each ADR. The evidence to trust is the gates, not the authorship: finite-difference checks on the measurement Jacobians, property tests, a golden snapshot of the benchmark, and CI. Since the first week of the repository, every change lands through a pull request that must pass CI and that the maintainer merges.

Scope and responsible use

This is a research and evaluation library built from public sources, synthetic data and two public recorded datasets that are fetched, never vendored. It contains no real sensor data or drivers, no radio-frequency model (a GNSS outage is a gap in a stream; spoofing tests inject offsets into the filter), and no guidance, targeting or platform integration code. It is not a navigation product and is not qualified for any platform. To the maintainer's knowledge nothing in it is classified or restricted, but that is not a legal determination: check your own obligations before using it in a regulated or operational setting. Details in Scope and responsible use; report a vulnerability through the security policy.

Documentation

The site is at telschow.github.io/contested-nav; every page is also a Markdown file under docs/.

If you want	Read
To run it	Getting started
The argument and the mechanism	Concepts, calibration
What the filter assumes, every config key, the result format	Model and interface contract
Evidence on recorded data	EuRoC, TUM VI, UrbanNav (real GNSS outage), where the outage error comes from, a real front end
Spoofing, multipath and fault classes	Honest returns, slow ramps, classification, multipath
How fast it is, measured	Performance

Layout

src/navkit/   estimators (ESKF, dead reckoning), fdir, sensors, degrade, eval, geometry, io
configs/      benchmark scenarios (configs/benchmark.yaml)
scripts/      thin wrappers, coverage ratchet, doc-table and link checks, secret scan
benchmarks/   timing harness for docs/PERFORMANCE.md
docs/         site pages, ADRs, figures, defense assessment, product management
tests/        unit, property, golden-snapshot and documentation consistency tests

Acknowledgements

The recorded GNSS and IMU data on the UrbanNav page is from the UrbanNav dataset (github.com/IPNL-POLYU/UrbanNavDataset). The authors ask for a citation of Hsu et al. (2023), NAVIGATION. UrbanNav: an open-sourced multisensory dataset for urban positioning. The authors state that the dataset files are MIT licensed. The data is fetched with navkit urbannav fetch and is not part of this repository. Only aggregate results are published here.

License

MIT. See LICENSE.