Breakwater

A red team for performance pull requests. Breakwater treats a "faster" change's benchmark as the suspect. A Nemotron adversary reads the diff and writes attacks, and every line of code it writes runs in a Nebius Token Factory Sandboxes microVM. The goal is to find the inputs where the speedup reverses or the behaviour silently changes. You get a verdict (BLOCK / REVIEW / PASS), minimal counterexamples as ready-to-commit pytest tests, deterministic instruction-count measurements, and a bill by model tier.

Entry for the Nebius x NVIDIA Global AI Hackathon, Coding & Agentic Engineering track. Built Oct 8–30, 2026.

For judges

Criterion	Claim	Check it in a minute
Technological Implementation	Three Nemotron tiers on Token Factory, each with explicit reasoning control, structured output and its own line on the bill. LLM-written code runs only in Sandboxes microVMs branched from a recorded image. Measurement is deterministic (cachegrind Ir). Adjudication must quote its evidence. Regression tests are verified in the sandbox before they're offered	llm.py, backends/contree.py, the receipt on any run page (tier bill, backend, image UUID), artifacts/day1/contree.json
Design	One command or a GitHub Action gives a verdict, counterexample cards with tests, a report page, a PR comment and a live view. A recorded run can be replayed. The judge box runs four real versions of pip's version parser in your browser	Demo site: index, a PR page, the Codeflash sub-site, "Try it"
Potential Impact	9/9 documented regressions in pip's version parser caught pre-merge, on Docker and again on Sandboxes (best no-LLM baseline 7/9). 4/10 merged AI-written speedups change behaviour (3 adjudicated likely-bug). One does 363× the work on valid input	docs/backtest.md, scripts/day1/ground_truth.py, the Codeflash sub-site
Quality of the Idea	Falsify rather than prove: the adversary tries to break the speed and equivalence claims, and honest baselines show what the model adds	Why below, "What Nemotron adds" in docs/backtest.md
Track fit	A coding agent that writes, runs and tests code in Token Factory Sandboxes	The live page's sandbox gauge; an image UUID on every number
Feedback	Specific entries, each with a reproduction and a suggestion	docs/feedback-log.md

Demo site: https://adamwhitlock1.github.io/breakwater/ (GitHub Pages). Video: [x].

What it found

AI-written speedups (Codeflash corpus). These are ten merged Codeflash-bot "Speed up …" PRs in seven repos. The run was on Token Factory with Sandboxes (artifacts/backtest/codeflash-v2, $0.88).

• json_repair #113, "Speed up method ObjectComparer.is_same_object by 109%". Base returns as soon as two dicts' key sets differ. Head checks only the lengths, then recursively compares values key by key. On same-sized JSON objects that differ by one key, head does up to 363× the instructions. Both versions return the same answer. The claimed path does get faster (0.48–0.58×).

• 4 of 10 change behaviour. Three were adjudicated likely-bug:

  • Math-Interpreter #1 ("by 11%") now tokenises Arabic-Indic digits as numbers. This is the same bug class as packaging #1082.
  • caddy-feeds #4 ("by 17,376%") added an lru_cache, so a list argument now raises TypeError.
  • patterns #12: a nested list raises TypeError instead of AttributeError.

  The fourth, rendercv #18, was adjudicated intended.

• The no-LLM baselines found nothing here. These repos have few tests to mine, so the Nemotron plan and search did all the finding.

Human-written speedups (pypa/packaging, the version parser inside pip). In 2026 it merged a string of performance PRs, and several changed behaviour:

PR	Claimed	What actually changed	Fixed by
#1082 fast path for simple versions	~2× faster parsing	Intermediate commit accepted "1. 0" and Arabic-Indic digits; merged version changed the exception type for oversized integers; slower on non-simple versions	review, then #1155 two months later
#1108 tuple instead of frozenset	faster SpecifierSet	SpecifierSet(">1.0.0,>1.0") != SpecifierSet(">1.0")	#1109, same day
#1116 overhaul _cmpkey	faster comparisons	Version objects pickled with 26.0 fail to load in 26.1	#1163 (issue #1162)

The backtest covers 28 merged 2026 perf PRs plus #1082's intermediate commit, each audited from its parent commit. Recall on the 9 verified regressions by arm:

Arm	LLM	Verified (9)
The PR's own tests and benchmarks	—	0 (by construction: these PRs merged green)
Plain Hypothesis (st.text())	no	2
Hostile mutations of the PR's own test strings	no	7
Ultra's planned example inputs only	Ultra	6
Breakwater (plan + Lightning strategies + search + mutations)	Ultra, Lightning	9

This is the beginning of the README. Read the rest at the original. Read the full README ↗