MUTINY

---

For judges

Looking for	Go to
The whole thing in under three minutes	the demo video: the problem, a live run, Tavily, Nemotron on Token Factory, and the census
How NVIDIA Nemotron is used	How NVIDIA Nemotron is used, then the measured comparison of all four variants
Where Token Factory accelerated the work	Where Token Factory accelerated the work, with doctor run against a live account
Other Nebius tools and services	Nebius Sandboxes, checkpoint forking, 40 isolated microVMs in 3.7s
Tavily	Was the old behaviour promised to anyone?, a real run quoting the documentation it contradicts
Does it actually work	Results: 266 merged "no behaviour change" pull requests from the top 300 PyPI packages, three regressions, one already reverted by its maintainer
AI review bots against execution	Opinion against observation: what Copilot, Sourcery, Cursor Bugbot and others said on the census PRs, every claim run on both commits
What it gets wrong	docs/findings-census.md, 8 of 27 census flags were its own mistake, classified by cause; docs/findings-limits.md, six classes of false finding and one public retraction
Agents that write, run and test code	An agent that checks its own work: a rewrite that repairs itself, and an MCP server coding agents call on their own edits
Running it yourself	Setup, macOS, Linux and Windows
It running on a real PR	the comment it left on pull request #1

---

A pull request says refactor. The tests pass. MUTINY runs the old and the new code on the same generated inputs, each in its own Nebius Sandbox, and shows you the exact input where they disagree -- or tells you there isn't one.

We pointed it at 266 merged pull requests from the 300 most-downloaded PyPI packages, every one of which said it changed nothing.

import google.genai as genai; genai.voices
    2.25.0             <module 'google.genai.voices'>
    2.26.0 ... 2.33.0  AttributeError: module 'google.genai' has no attribute 'voices'

That came from a pull request titled "chore: expose GenAI domain types", and it is still in the latest release. It is one of three:

regressions found	3: werkzeug (the maintainer reverted it five days later: "flask uses it"), ipython (shipped in four releases, fixed 106 days later), google-genai (above, still unreported)
AI review bots on those three pull requests	none
placebo: 224 PRs whose code cannot have changed, flagged	0
its own mistakes	8 of the 27 PRs it flagged, each listed with its cause
cost	$4.15 for all 490, Nemotron 3 on Nebius Token Factory

The list was frozen and committed before anything ran on it, every divergence was read by hand against its diff, and each regression was reproduced on the commits and, where it shipped, on the published wheels (the census).

And the AI reviewers? Copilot, Sourcery, Cursor Bugbot, Codex and six others reviewed 42 of the 266 -- none of the three regressions. Where they did comment, we ran their claims too: two of them were real breakages that shipped, in isort and pyjwt, on code MUTINY could not reach. Reading a diff and running it catch different things, and the census shows which.

It works on agent-written code the same way. This is a refactor Nemotron 3 Super produced for cachetools.Cache.__setitem__; it reads as a tidy-up and breaks the library for every caller who does not pass getsizeof:

c = Cache(maxsize=2); c["a"] = 1; sorted(c.items())
    before: [('a', 1)]
    after:  AttributeError: '_DefaultSize' object has no attribute 'get'

Why this exists

Published work finds 19–35% of LLM-generated refactorings are functionally incorrect, and over 21% of those errors pass the project's existing tests (Dristi and Dwyer, 2026). We reproduced the first half independently: of 18 refactors Nemotron produced for real functions in semver, cachetools and packaging, 17% were rejected by the repositories' own test suites.

Teams are now merging agent-written changes at a rate no review process was designed for, and the test suite is the only gate. When the change is subtle — an edge case, a default argument, an eviction order — the suite is not enough.

Prior art, and what this adds

The core idea is not new, and it would be a worse project if it pretended to be. Testora (Pradel, ICSE 2026) runs the old and new version of a pull request on LLM-generated inputs and uses the PR's own text to separate intended differences from regressions. Mokav searches for an input that exposes a difference between two program versions. Meta's just-in-time catching tests write tests aimed at a diff, and report how many of their catches were false: the cost of letting a model write the assertion. No commercial review tool we found executes both versions; they read the diff.

What MUTINY adds is what it took to make that idea safe to point at a stranger's code and quiet enough to leave on:

• Isolation that scales. Every probe runs in a microVM forked from one warm Nebius Sandboxes checkpoint: 40 forks in 3.7 s, so a pull request that hangs, segfaults or deletes files costs nothing.
• Documentation as a second witness. Tavily finds whether the old behaviour was promised, and a quote is only shown if it appears verbatim on a page that belongs to the library under review.
• Objects that take a paragraph to build. When no generated input can construct the receiver, a loop reads the class, writes a recipe and proves it by running it, before any probe is written.
• Every way it has been wrong, written down. Six causes of false alarms and one false negative, each with the case that exposed it, the fix, and a test -- and, from the census, the 8 of 27 flags on real pull requests that were its own mistake, sorted into three causes with the fix each argues for.
• It runs where code gets written. A GitHub Action that comments only on undescribed changes, an MCP server a coding agent calls on its own edits, a self-repair loop, and a hosted demo.

How it works

The model's only job is to produce inputs. Execution produces the evidence.

  function under review
          │
          ├── Nemotron generates call expressions and stateful scenarios
          │   (informed by the diff, by tests that already cover the line,
          │    and by the concrete subclasses that can stand in as receivers)
          │
          ├── run every input against the ORIGINAL
          ├── run every input against the REWRITE
          │
          └── compare outcomes ─── divergence? here is the input that proves it

That division of labour is the whole design, and it is why this works where our first attempt did not. A wrong assertion manufactures a false finding. A wrong input is rejected identically by both versions and simply contributes nothing. Bad inputs are wasted, never wrong — so the model only has to be productive, not correct.

An earlier version of this project asked Nemotron to write a test proving a specific bug. It managed that 13% of the time at 120B and 13% at 550B. Inverting the burden moved the same task to 62–67%.

An agent that checks its own work

The track this was built for asks for agents that write, run and test code. The failure such an agent is prone to is the one this project measures: it rewrites something, the tests pass, and it reports the change as safe because nothing it can see says otherwise. So MUTINY closes that loop in two ways.

It can make a rewrite repair itself. mutiny verify <repo> <function> --repair N has Nemotron rewrite a function, runs both versions, and when they disagree hands the model the exact inputs it got wrong -- as execution results, not a reviewer's opinion -- and runs its fix again on those inputs and on fresh ones. A run on cachetools.Cache.__setitem__, recorded from the command line:

attempt	what Nemotron wrote	what running both versions showed
1	old_size = self.__size.get(key, 0)	12 of 14 inputs disagree: AttributeError, because the default size map has no .get
2	old_size = self.__size[key] if key in self.__size else 0	13 of 22 disagree: every one a timeout. The map defines __getitem__ but not __contains__, so in iterates it by index, forever
3	the original if/else, with diffsize renamed to needed	0 of 20 disagree, including every input that broke attempts 1 and 2

Both broken attempts read like reasonable tidy-ups, and both would have passed a review that only read them. The hosted demo does not replay this one: it only replays runs recorded on the deployment, and when we tried to record it there, nine fresh rewrites in a row came out correct. The second is also the bug that taught MUTINY to time out each probe on its own -- below. And the third is the honest end of the story: the improvement the model could prove safe was a rename.

It is a tool a coding agent can call on itself. mutiny mcp serves it over the Model Context Protocol, so Claude Code, Cursor, Codex or Copilot can verify their own uncommitted edits before claiming they are safe:

pip install "mutiny-verify[mcp] @ git+https://github.com/TusharTechs/mutiny"
claude mcp add mutiny -e NEBIUS_API_KEY=... -e NEBIUS_PROJECT_ID=... -- mutiny mcp

verify_working_copy compares the working copy against a base revision, uncommitted changes included, and takes the agent's own statement of intent so differences it announced are kept apart from ones it did not. Called over stdio on an uncommitted "readability refactor, no behaviour change" of the same method:

{
  "behaviour_changed": true,
  "witnesses": [{
    "call": "c = LFUCache(2); c['a'] = 'x'; c['b'] = 'yy'; c['a'] = 'zz'; (sorted(c.items()), c.currsize, len(c))",
    "before": "([('a', 'zz'), ('b', 'yy')], 2, 2)",
    "after": "AttributeError: '_DefaultSize' object has no attribute 'get'"
  }],
  "announced_by_intent": false,
  "seconds": 106.2, "cost_usd": 0.0069
}

Both tools are annotated read-only. Neither touches the agent's repository, and both versions run in Nebius Sandboxes rather than on the agent's machine.

Results

Measured on real code, not fixtures. Every number is reproducible from experiments/.

266 pull requests that said they changed nothing -- merged pull requests from the last twelve months, up to twelve per repository and newest first, across the 190 suitable repositories of the 300 most-downloaded PyPI packages, whose title or description claimed refactor, cleanup, simplify, perf or "no behaviour change". The list was committed before anything ran on it, so it cannot have been tuned to the results (full write-up):

reached a verdict	126 of the 253 with behaviour to check
behaviour changed	36
flagged as changed and not described in the PR	27
of those, a real undescribed change, checked by hand	10
of those, regressions	3
of those, the tool's own mistake	8, each classified with the fix it argues for
placebo: 224 PRs whose code cannot have changed, flagged	0
cost	$4.15 for all 490, under a cent each

The three regressions, each reproduced by hand:

Pull request	Titled	What happened next
pallets/werkzeug#3333	"refactor exceptions"	renamed default_exceptions with no alias; the maintainer reverted it five days later: "still private, but flask uses it"
ipython/ipython#15287	"refactor: centralize image format handling"	real WebP images reported as PNG in four releases (9.16.0 to 9.17.1); fixed upstream 106 days after merge
googleapis/python-genai#3005	"chore: expose GenAI domain types ..."	genai.voices raises AttributeError from 2.26.0 through the latest 2.33.0; still unreported

Plus a behaviour change more-itertools shipped in 11.2.0 with no changelog line, and a bug starlette fixed without noticing. One flag in three was a real change nobody had written down; three in ten were MUTINY's mistake, and every one is listed.

Agent refactor safety — Nemotron refactors a function, the repository's own tests judge it, and the differential runs regardless:

refactors produced	18 / 21 functions
functionally broken (tests rejected them)	17%
of those, differential also caught	67%
of the 14 tests accepted, behaviour changed	0
cost	~$0.01 per function

Detection on real bug-fix commits — 24 commits across three libraries, each run twice, ground truth taken from commit messages:

behaviour changes detected	62%
behaviour-preserving commits falsely flagged	0
runs agreeing across repeats	94%

Across those 14 correct refactors and 5 behaviour-preserving commits, MUTINY flagged none. That zero is narrower than it reads: it holds for code that did not change, and it held again on 224 placebo pull requests in the census above. On code that did change, the census found the tool's own mistakes in 8 of the 27 pull requests it flagged.

Merged pull requests nobody chose for us -- 49 recent merged PRs from twelve widely used libraries (full write-up):

had behaviour to check	45 of 49
reached a verdict	40 of 45
behaviour changed	12 in the recorded run, 10 of them announced in the PR's own title; 2 more in arrow once object construction reached them
commented on	the 2 that were not: jd/tenacity#679 and the PR stacked on it
cost	$0.39 for all 49, under a cent each

Opinion against observation

Every AI review bot answers "did this change behaviour?" by reading the diff. To compare that with running it, Nemotron was asked to review the 13 pull requests where execution found a behaviour change -- reasoning on, the title, description and full Python diff -- and to name a concrete call whose result differs before and after. Then every call it named was executed on both versions, through the same sandboxes, canonicalisation and three-sample confirmation as MUTINY's own witnesses (experiments/review):

reviewer	calls it named	held when run	behaved identically on both versions	could not run
Nemotron 3 Super 120B	32	21	7	4
Nemotron 3 Ultra 550B	32	15	9	8

Reading the diff works well on fixes that say what they fix: both models were right on every call for cachetools#413 and shortuuid#115. It fails where the question matters. On jd/tenacity#679, titled "refactor: drop always-true truthiness checks", both models concluded behaviour was preserved, restating the author's argument that wait "is typed such that it can never be falsy". The type annotation says what callers should pass. Execution shows what the code accepts: Retrying(wait=None)._run_wait(state) returned 0.0 before and raises TypeError after.

Two smaller results. The 550B model did worse than the 120B one here too. And as a check on the harness rather than the reviewers, MUTINY's own corpus witnesses were run through it as well: they reproduced on 11 of 12 pull requests. The twelfth, tenacity#645, did not reproduce in this harness for either side, and is counted against nobody.

Getting there meant finding three separate ways a comparison can lie: memory addresses in the default repr, which differ every process; set and dict iteration order, which made identical collections look different; and hash-seed randomisation, which changes the behaviour of anything whose logic touches dict ordering. Each produced confident, specific, entirely false findings before it was fixed. A verification tool that cries wolf gets switched off in a week.

The review bots projects actually install, on the census. Copilot, Sourcery, Cursor Bugbot, the Codex connector and six others reviewed 42 of the 266 census pull requests, and every behaviour claim they made that could be run was run on both commits or the published wheels (botcheck.md):

census PRs with any AI review	42 of 266
of the 36 where execution found a divergence	5
of the 3 regressions execution found	0
bot claims that held when run, on PRs MUTINY could not probe	3 (isort#2509 and pyjwt#1136 shipped; redis-py#3994)
bot claims that did not hold on the merged code	2

The bots caught what MUTINY cannot reach: isort 9.0 changed what sort_file returns, and pyjwt 2.12.0 could not be imported on Python 3.9 without an undeclared dependency, fixed in 2.12.1 the next day. Execution caught what no bot was there to read. They are complements, and the census says so in numbers.

How NVIDIA Nemotron is used

Nemotron 3 Super does every model job in the pipeline, and none of them is allowed to produce the evidence:

1. It writes the code under review. mutiny/refactor.py asks nemotron-3-super-120b-a12b to refactor a real function for readability while preserving behaviour, when there is no pull request to check.
2. It writes the inputs. mutiny/inputs.py asks it for call expressions and stateful scenarios that reach the changed lines, steered by the diff, by the tests that already cover those lines, and by the concrete subclasses available as receivers.
3. It builds objects that are hard to construct. When no input runs, mutiny/construct.py has it read the class and write a recipe, which is kept only once running it succeeds.
4. It explains a witness in one sentence, with reasoning off.
5. It judges whether the change said so, against the pull request's own title and description, with reasoning on.
6. It reads the documentation Tavily found and must quote one sentence word for word, which is then checked against the page (mutiny/contract.py).

Nemotron 3 Ultra is the fallback when Super returns nothing.

Choosing a model is about task shape, not parameter count

We measured all four on an identical prompt asking for 45 call expressions:

model	usable inputs	completion tokens	reasoning emitted	time
Nemotron 3 Nano 30B-A3B	44	9,846	26,300 chars	33.9 s
Nemotron 3.5 Lightning	1	14,000	0	56.0 s
Nemotron 3 Super 120B-A12B	42	4,520	0	10.7 s
Nemotron 3 Ultra 550B-A55B	—	—	—	—

Super emits no reasoning at all on this task and answers directly, in a third of Nano's time for half the tokens. Nano deliberates for 26,000 characters about how to write call expressions and frequently exhausts its allowance before writing one.

The reverse held for the harder task. On writing a test that mechanically proves a specific bug, Ultra scored identically to Super — 2/15 each, and Super failed worse than the smaller models when starved of tokens. Bigger was not better; the shape of the work decided the model.

Reasoning on or off is a per-task decision, measured

Nemotron 3 reasons by default, and the trace is billed. It can be switched off per request with chat_template_kwargs={"enable_thinking": False}. Whether to do that was measured on the 12 witnessed behaviour changes, for both of the calls MUTINY makes after finding one (experiments/review/reasoning.py):

call	reasoning off saves	and does
plain-English summary of the witnesses	about 6x the tokens, 2.5x the time	summaries at least as accurate
"did the pull request describe this?"	about 8x the tokens	flips 7 of 12 to "not described" -- including fixes whose titles say exactly what changed

So it is off for the summary and on for the judgement. Turned off everywhere, the cheaper tool would cry wolf on most real fixes -- the one failure a review bot cannot survive.

Where Token Factory accelerated the work

mutiny doctor against a live account: Token Factory credentials, the project Sandboxes routes by, and the spend ledger split by model. 1,080 billed calls and 2,446 more served from cache, for $4.69 across every experiment in this repository.

• One OpenAI-compatible endpoint for four models. Swapping Nano for Super for Ultra is a string change, which is what made the comparison table above cheap enough to actually run rather than assume.
• mutiny/models.py wraps it with a spend cap, a persistent token ledger and content-hash response caching. Re-running an experiment costs nothing, which turned a 2.5-hour benchmark into a 20-minute one and made iterating on the measurement affordable. Every experiment before the census came to $4.69 (the ledger above); the census added $4.15 for 490 pull requests.
• Reasoning-aware retry. Nemotron 3 bills its reasoning trace against max_tokens, so a generous-looking allowance can be spent entirely on thinking. The client detects that and widens, except for one signature where widening provably never helps (see docs/feedback.md).

Nebius Sandboxes

mutiny/sandbox.py implements the executor against the ConTree SDK, behind the same interface as the local one.

The fit is unusually good: image.run() returns a new image rather than mutating the old one, so running N probes against one warm image is N forks from a single checkpoint, isolated by construction. That is exactly this workload's shape — one expensive setup (clone, install, warm the interpreter), then hundreds of short independent executions that must not see each other's state. Locally we pay setup once and then serialise; on Sandboxes the batches run concurrently, up to the documented ceiling of 50.

Measured against the live service, on python-semver installed from source:

warm checkpoint (upload, extract, pip install -e .)	9.8 s, paid once
40 forks, run sequentially	50.4 s
40 forks, run concurrently	3.7 s — 0.09 s each
local subprocess, same probes	0.10 s each

Forks are genuinely isolated: each inherits everything the checkpoint wrote, sees nothing a sibling wrote, and leaves the checkpoint unchanged. Sandbox and local execution agreed on 23 of 23 observations.

The honest comparison is that a single sandbox fork is slower than a local subprocess — 1.2 s against 0.1 s, being network and a microVM. Concurrency is the entire reason to be here, and it inverts that: at 40 forks the per-fork cost is below local, and the work is spread across forty isolated machines rather than competing on one. That matters twice over, because the code being executed was written by a model and should not run on a developer's laptop at all.

Try it

mutiny verify path/to/repo Cache.__setitem__

Nemotron rewrites the function, both versions run on generated inputs inside forks of one warm Sandboxes checkpoint, and you get either the input where they disagree or a clean bill:

cachetools  src/cachetools/__init__.py  Cache.__setitem__  (cachetools)

rewriting with nemotron-3-super-120b-a12b...
    -        diffsize = size - self.__size[key]
    +        old_size = self.__size.get(key, 0)
    +        needed = size - old_size
    ...
warming a sandbox checkpoint...
  ready in 8.8s, 44 KiB uploaded
generating probes...
  10 probes
running both versions...

Behaviour changed.  5 of 10 probes disagree.

  c = LRUCache(3); c["a"] = "x"; c["b"] = "yy"; (sorted(c.items()), c.currsize, len(c))
    before: ([('a', 'x'), ('b', 'yy')], 2, 2)
    after:  AttributeError: '_DefaultSize' object has no attribute 'get'

36.2s, $0.0065

--local runs the probes on this machine instead, which is faster for a single pass and appropriate only for code you trust.

Reviewing a change that already exists

mutiny verify-diff path/to/repo --base main --head my-branch

Finds the functions a branch touched and checks each one, comparing the two revisions. Nothing is checked out — git archive and git show read straight out of the object store, so your working copy is untouched and CI jobs sharing a clone do not fight each other. One checkpoint is warmed at the merge base, and each fork has the head version of the changed files laid over the top.

python-semver  d8813b67^..d8813b67
2 changed function(s) to check, 1 skipped

Version.next_version  src/semver/version.py
  behaviour changed — 1 of 14 probes disagree
    str(Version(1,2,3, prerelease="rc").next_version("prerelease"))
      before: '1.2.3-rc'
      after:  '1.2.3-rc.0'

2 of 2 changed functions behave differently.
28.4s, $0.0070

Point it at your own code

mutiny verify-url https://github.com/owner/repo/pull/123
mutiny verify-url https://github.com/owner/repo

A pull request is checked against its merge base, so other people's work on main is not attributed to the change under review. A repository with no change to review has its most heavily branched function rewritten, and that rewrite checked.

Only public GitHub repositories, the clone is size-capped, and nothing is installed or executed on the machine running MUTINY — that happens inside a sandbox, which is the reason the sandbox is there.

The web interface

The hosted demo is mutiny-verify.vercel.app, and it is the same engine:

uv pip install -e ".[web]" --python .venv/bin/python
.venv/bin/python -m uvicorn app.server:app --port 8000

• Recorded runs. Each card replays a run recorded on the deployment, in about ten seconds, labelled as a recording, with the live run one click away.
• Any public pull request, live. The URL box fetches the change, installs both versions in a sandbox and streams the run. Every input appears as one cell of a strip, green where the versions agreed and red where they did not, and the run ends with what to do: a review comment carrying the witness, and the two mutiny pin commands.
• A guided tour drives a recorded run and points at each part as it appears.
• The census wall draws all 490 census pull requests as cells, coloured by the hand check, with the ones an AI review bot commented on ringed on request.

A public endpoint that installs arbitrary code is an obvious target, so it has limits: a shared spending ceiling, a per-run cap, a per-address rate limit, and the code only ever runs inside a sandbox. A judge code, if one is configured, lifts the rate limit (docs/deploy.md).

Setup

Python 3.11 or newer. No other system dependency: MUTINY talks to GitHub over HTTPS, so a local git is optional, and the code under review is installed and executed inside a Nebius sandbox rather than on your machine.

macOS and Linux

git clone https://github.com/TusharTechs/mutiny.git
cd mutiny
python3 -m venv .venv
.venv/bin/python -m pip install -e ".[dev]"
cp .env.example .env            # then add NEBIUS_API_KEY and NEBIUS_PROJECT_ID
.venv/bin/python scripts/doctor.py

Windows (PowerShell)

git clone https://github.com/TusharTechs/mutiny.git
cd mutiny
py -3 -m venv .venv
.venv\Scripts\python -m pip install -e ".[dev]"
Copy-Item .env.example .env     # then add NEBIUS_API_KEY and NEBIUS_PROJECT_ID
.venv\Scripts\python scripts\doctor.py

If you use uv, uv venv --python 3.12 .venv and uv pip install -e ".[dev]" do the same thing faster. It is not required.

doctor.py checks connectivity, credentials, model availability and remaining spend, and prints what is wrong rather than failing silently. Run it first whenever anything misbehaves.

Then, on any platform:

mutiny verify-url https://github.com/python-semver/python-semver/pull/401

Reproducing the results

The benchmarks clone real repositories and run against their history. Paths below use the POSIX form; on Windows substitute .venv\Scripts\python.

.venv/bin/python -m pytest tests/ -q             # 119 tests, no network needed
.venv/bin/python experiments/refactor/run.py     # agent refactor safety
.venv/bin/python experiments/diff/scaled.py 8    # detection on real fix commits
.venv/bin/python experiments/prs/run.py          # the 49 merged pull requests
.venv/bin/python experiments/census/run.py both  # the census, from the committed list
.venv/bin/python experiments/census/bots.py      # AI review-bot comments on the census

experiments/census/collect.py rebuilds the census list from scratch (about 3,000 GitHub API calls, so it needs GITHUB_TOKEN); run.py only executes the committed manifest.json and resumes where it stopped.

experiments/prs/run.py caches GitHub API responses under experiments/prs/cache/, which is not committed. The first run repopulates it; set GITHUB_TOKEN to lift the 60 requests per hour anonymous limit.

Architecture

A URL goes in. Two versions of the same code come out the other side, running on the same generated inputs inside the same sandbox image, and the answer is either an input where they disagree or a statement that none was found.

%%{init: {"flowchart": {"useMaxWidth": true, "nodeSpacing": 26, "rankSpacing": 34, "padding": 6}}}%%
flowchart TB
    URL(["<b>GitHub URL</b>"])
    REMOTE["<b>remote.py</b><br/>two revisions over HTTPS"]
    WHAT{"pull request?"}
    PR["<b>review.py</b><br/>merge base vs head"]
    RW["<b>refactor.py</b><br/>Nemotron rewrites it"]
    WARM[["<b>Nebius Sandboxes</b><br/>install once · 9.8s"]]
    GEN["<b>inputs.py</b><br/>Nemotron 3 Super writes probes<br/>via Token Factory"]
    BUILD["<b>construct.py</b><br/>read the class · build a recipe<br/>prove it by running it"]
    BEFORE["<b>before</b>"]
    AFTER["<b>after</b>"]
    CMP["<b>differential.py</b><br/>canonicalise · compare"]
    CONF{"reproduces?"}
    EXP["<b>explain.py</b><br/>one-sentence summary<br/>reasoning off"]
    SAID{"did the change<br/>say so?<br/>reasoning on"}
    DOC["<b>contract.py</b><br/>Tavily search + extract<br/>verbatim quote, checked"]
    OUT(["<b>Witness</b>"])
    CI["<b>ci.py · report.py</b><br/>one PR comment, edited in place"]
    PIN["<b>pin.py</b><br/>a test that pins what it saw"]
    QUIET(["<b>Reported as doing<br/>what it says</b>"])
    NONE(["<b>No divergence</b>"])

    URL --> REMOTE --> WHAT
    WHAT -- yes --> PR --> WARM
    WHAT -- "no, so make one" --> RW --> WARM
    WARM --> GEN
    GEN -- "nothing ran" --> BUILD -- "now it can" --> GEN
    GEN -- "inputs dealt across<br/>8 microVM forks a side" --> BEFORE --> CMP
    GEN --> AFTER --> CMP
    CMP -- "they differ" --> CONF
    CMP -- "all agreed" --> NONE
    CONF -- "flaky, not evidence" --> NONE
    CONF -- yes --> EXP --> SAID
    SAID -- yes --> QUIET
    SAID -- "no, or not said" --> DOC --> OUT
    OUT --> CI
    OUT --> PIN

    style URL fill:#4ecdc4,stroke:#2a9d94,color:#000
    style WARM fill:#e879f9,stroke:#c026d3,color:#000
    style GEN fill:#fbbf24,stroke:#d97706,color:#000
    style BUILD fill:#fbbf24,stroke:#d97706,color:#000
    style CONF fill:#fbbf24,stroke:#d97706,color:#000
    style SAID fill:#fbbf24,stroke:#d97706,color:#000
    style DOC fill:#8b5cf6,stroke:#6d28d9,color:#fff
    style OUT fill:#e56a6a,stroke:#b83c3c,color:#000
    style QUIET fill:#8b97a8,stroke:#5d6878,color:#000
    style NONE fill:#5ac77e,stroke:#2f9350,color:#000

The one idea this is built on

Inputs don't need to be correct. Assertions do.

Ask a model to write a test and it must produce an assertion — a claim about what the right answer is. Get that wrong and you have manufactured a false finding out of nothing.

Ask a model to write an input and it cannot lie. A bad input is rejected identically by both versions and contributes nothing. A good one is executed twice, and the two results are compared by a machine that has no opinion about which is correct.

That inversion is the whole design. The model generates candidates; execution produces the evidence. It is also why this works at all — an earlier version that asked Nemotron to produce proofs scored 13% at both 120B and 550B, and no amount of model size fixed it.

Why there are two revisions, not one

Input	before	after
Pull request URL	the merge base	the head commit
Repository URL	the code as written	Nemotron's rewrite of it
mutiny verify-diff	any ref	any ref

Comparing against the merge base rather than the base branch tip matters: other people's work on main is not this author's change, and attributing it to them produces findings that waste a reviewer's time.

Nondeterminism is the adversary, not the bug

The first version of this reported findings that were real divergences and completely worthless. Five separate causes, each found by chasing a confident, specific, false result:

Cause	What it looked like	Fix
Memory addresses	every object without __repr__ "changed"	strip at 0x...
Set and dict order	identical frozensets, different repr	sort before comparing
Hash seed	a real-looking cache eviction bug	pin PYTHONHASHSEED
random	RRCache evicts differently each run	seed it
os.urandom	uuid4 differs by construction	require self-consistency

The last one is the general answer and the reason confirm exists: before a divergence is reported, the same version is run twice on the same input. If it disagrees with itself, the probe is discarded — it was never evidence. One of these got as far as being called "the finding this project exists for" before it turned out to be hash-seed randomisation; that retraction is written up in docs/findings-differential.md.

The worst bug was a silence

Every cause above is a false alarm. The most dangerous bug this project has had was the opposite: a change that broke something, reported as safe.

It surfaced in MUTINY's own self-repair demo. Nemotron's first rewrite of cachetools.Cache.__setitem__ broke on 12 inputs; given those inputs, it produced a "fix" using key in self.__size. On the default size map, which defines __getitem__ but not __contains__, Python falls back to iterating it by index -- forever. Every probe on the repaired version hung, every after-side sandbox fork hit its operation timeout, every observation in those forks was lost, and compare() only compares inputs it has seen on both sides. The run printed "preserved -- 0/25 probes ran on both versions and agreed", then "Repaired on attempt 2."

Two changes close it. Each probe now has its own ten-second allowance inside the sandbox, raised as a BaseException so a bare except Exception in the code under test cannot swallow it: a hang is recorded as an outcome and compared like any other. And nothing is called preserved or repaired unless every input was observed on both sides -- an input that produced no result is counted and reported as not verified, an absence of evidence rather than agreement. When the model produced the same hanging "fix" again on a later run, it was caught as 13 of 22 probes disagreeing, every one a timeout, and handed back to the model like any other failure -- attempt 2 in the repair run above.

What Sandboxes make possible

Installing a real repository takes tens of seconds. Doing that once per probe would make the whole approach unusable, which is the practical reason differential verification is not already a common technique.

A Nebius Sandbox checkpoint is installed once and then forked. Forty forks run in 3.7s against 50.4s sequentially, and the forks are microVMs rather than threads — so a probe that segfaults, hangs, or deletes a file takes nothing with it. Executing a stranger's pull request is the entire point, and it never touches the server.

	measured
Warm a checkpoint	9.8s
40 forks, concurrent	3.7s
40 runs, sequential	50.4s
Django — 2932 files, end to end	83s
Cost of one verification	under a cent on average ($4.15 for the census's 490)

Where each model earns its place

Model choice here is decided by the shape of the task, not by parameter count.

Task	Model	Why
Generating probes	Nemotron 3 Super 120B	42 usable inputs in 10.7s with zero reasoning tokens. Nano spent 26,300 reasoning characters to produce 44; Lightning produced 1.
Rewriting a function	Nemotron 3 Super 120B	Needs to produce plausible, compiling code — not to be right
Building a receiver	Nemotron 3 Super 120B	A recipe is only kept once running it succeeds, so a wrong one costs a retry, never a finding
Explaining a witness	Nemotron 3 Super 120B, reasoning off	Grounded in evidence already on the page; the prompt forbids speculating about intent or correctness
Judging the PR's own description	Nemotron 3 Super 120B, reasoning on	Turning reasoning off flipped 7 of 12 judgements to "not described"
Reading the documentation	Nemotron 3 Super 120B	Must quote a sentence verbatim; the quote is checked against the page Tavily extracted
Fallback on empty reply	Nemotron 3 Ultra 550B	Super uniquely returns empty content and empty reasoning under token pressure — reported in docs/feedback.md

The modules

module	responsibility
mutiny/session.py	the verification loop, yielded as events
mutiny/differential.py	runs both versions, canonicalises observations, compares
mutiny/sandbox.py	warm checkpoint, forks, archive upload
mutiny/inputs.py	Nemotron generates probes; execution-validated before use
mutiny/scenarios.py	where receiver objects come from: tests, subclasses, recipes
mutiny/construct.py	Nemotron writes a recipe for an object nothing else can build, kept once it runs
mutiny/contract.py	Tavily search and extract, then a verbatim quote checked against the page
mutiny/remote.py	GitHub over HTTPS — tarballs, merge base, no git binary
mutiny/review.py	resolving a change into changed functions
mutiny/refactor.py	Nemotron rewrites a function; applied in place
mutiny/explain.py	Nemotron summarises the witnesses and judges whether the change said so
mutiny/report.py · mutiny/ci.py	the pull request comment, and the GitHub Action that posts it
mutiny/pin.py	turns a witness into a test the repository keeps
mutiny/mcp_server.py	MUTINY as an MCP tool a coding agent calls on its own edits
mutiny/diff.py	changed lines, source-vs-test filtering, enclosing functions
mutiny/coverage.py	which tests execute which line — steers probes and test selection
mutiny/source.py	locating a function by qualified name, showing just enough of it
mutiny/models.py	Token Factory client: per-run spend cap, ledger, caching, retry
mutiny/budget.py	what a public deployment is allowed to spend
mutiny/fetch.py	git clone, for local experiments that have git

One engine produces the events; the CLI and the browser are two renderings of it. There is no second implementation to drift.

An observation records the value's type, a canonicalised repr, and for failures the exception type and normalised message. Memory addresses are stripped, unordered containers are sorted, and the hash seed is pinned — because an observation that varies for reasons the caller cannot control is not evidence.

Run it on every pull request

Review happens in the pull request. A tool that lives anywhere else is one somebody has to remember, and a tool that usually has nothing to say is one they stop remembering.

# .github/workflows/mutiny.yml
name: MUTINY
on: pull_request
permissions:
  contents: read
  pull-requests: write

jobs:
  behaviour:
    runs-on: ubuntu-latest
    steps:
      - uses: TusharTechs/mutiny@main
        with:
          nebius-api-key: ${{ secrets.NEBIUS_API_KEY }}
          nebius-project-id: ${{ secrets.NEBIUS_PROJECT_ID }}

It says nothing on most pull requests, and that is the point. Of 49 merged pull requests measured in docs/findings-pull-requests.md, twelve changed behaviour and ten of those were changes the author had already announced in the title. Reporting those is noise wearing the costume of diligence. The comment appears when a behaviour difference is one the change does not mention:

This change alters behaviour it does not mention

max_ver

5 of 11 generated inputs produce a different result before and after this change.

max_ver now raises TypeError whenever either argument is already a Version instance, because it calls Version.parse on it. Previously the function accepted both strings and Version objects.

max_ver(Version.parse("1.0.0"), Version.parse("1.0.0"))
# before:  '1.0.0'
# after:   TypeError: not expecting type '<class 'semver.version.Version'>'

That one is real and merged. python-semver#401 is titled "Simplify max_ver and min_ver", and its description says it uses max() and min() to make the two functions consistent. The code it replaced had an explicit elif not isinstance(ver1, Version) branch — it accepted Version objects deliberately. The one-line replacement does not.

Two rules, both about trust rather than capability:

• It never fails a build. A behaviour difference is information a reviewer weighs, not a verdict. A bot that blocks merges on its own judgement is switched off within a week.
• One comment per pull request, edited in place. Pushing a fix replaces the warning rather than leaving it standing above a correction nobody scrolls to.

input
comment-on	undescribed (default), any, or never — never still writes the run summary
probes	inputs generated per function, default 20
max-functions	changed functions to check, default 6
cap-usd	ceiling on model spend for one run, default 1.00

A run costs under a cent on average: $4.15 for all 490 in the census.

Was the old behaviour promised to anyone?

"Behaviour changed" in an undocumented internal helper is a curiosity. The same difference in something the project's own documentation specifies is a promise about to be broken, and that is a different conversation in review.

That question is answered by text outside the repository — published documentation, changelogs, issue threads — so Tavily fetches it, and the comment escalates only when the documentation describes the behaviour being removed:

That is a real run on python-semver#401. The behaviour change is found by execution; Tavily then finds the page that promised the old behaviour, and the sentence "Returns the greater version of two versions strings." is quoted with the URL it came from.

The obvious failure mode of asking a model about retrieved text is that it paraphrases something the page does not say, and attaching a link makes that worse rather than better: it dresses an invention as evidence. So the model must return a sentence copied verbatim, and that sentence is checked against the page it came from. If it is not there, the finding is dropped entirely — the same rule as everywhere else here, that a claim which cannot be checked against something observed is not reported.

Set TAVILY_API_KEY to enable it. Without it the step is skipped and the witness is exactly as strong as it was; a search that fails or finds nothing costs the report nothing, because this escalates a finding and is never a dependency of one.

Keeping what it found

A finding is a moment. Somebody reads it, decides the new behaviour is what they meant, merges — and nothing stops the next refactor from moving it back. The comment carries a test that pins what was observed, and mutiny pin writes one directly:

mutiny pin --url https://github.com/owner/repo/pull/123 --side after

"""Behaviour pinned by MUTINY from https://github.com/owner/repo/pull/123."""
from mutiny.diff import _is_source


def test_is_source_my_setup_py():
    assert repr(_is_source('my_setup.py')) == 'False'

This does not break the rule that a model never writes assertions. It doesn't write these either: the input was generated, executed against both versions, and kept only because the two disagreed — and a human chose which side was correct before anything was written down. Nothing here is predicted. A value that cannot be reproduced faithfully, such as a canonicalised set or an object identity, is refused rather than pinned to something that would fail for the wrong reason.

What it does not do

• Side effects are invisible. A function that mutates its argument or writes a file and returns None gives nothing to compare.
• Exception chaining is not observed. One known miss is a refactor that changed __context__ while preserving the raised type and message.
• Pinning the hash seed hides seed-dependent behaviour. Necessary to compare two runs at all, and a real trade-off: it cost us a finding we had to retract.
• Rich domain objects are hard to probe. Everything found so far takes primitives or simply-constructible values. A function wanting three populated domain objects and a timezone-aware datetime defeats the generator, and yield collapses to zero.
• Unreachable code stays unreached. A branch requiring a particular OS, a network condition or a race has no call expression that gets to it.
• A generated input can break the type's contract. A stand-in object missing an attribute every real one has, or a hand-set private field, makes the new version fail where no real caller could. Four of the census's eight false flags were this.
• A probe's own setup can fail. When a refactor removes a module-level import or moves a name behind a deprecation alias, a probe that names it fails before reaching the function. Three more, and the eighth compared hash() of an object hashed by its memory address.
• "Not described" can be wrong. The intent judgement missed a description written in other words four times in the census. The witness is the evidence; the summary and the judgement are reading aids.
• No input could be built for 98 of 253 census pull requests. That is reported as coverage, never as "preserved".

docs/findings-differential.md records the measurements and the retraction in full, including a finding we reported internally as the headline result and then withdrew when a pinned hash seed showed it was an artifact of our own harness.

Licence

Apache-2.0.

AI disclosure

This is a tool for checking code a model wrote, so it says plainly which AI was used, where, and for what.

Building MUTINY

Tool	What it was used for
Claude Code (Anthropic)	Pair programmer throughout: most of the implementation and debugging, the experiment and census scripts, first drafts of the census hand check and of the documentation, the web page, and the demo video's production scripts (scene animation, timing, captions, mixing)

Inside MUTINY, at runtime

Tool	What it does
NVIDIA Nemotron 3 Super 120B, on Nebius Token Factory	Writes the inputs, rewrites a function when there is no pull request, builds hard-to-construct objects, summarises a witness, judges whether the change said so, and quotes the documentation
NVIDIA Nemotron 3 Ultra 550B, on Nebius Token Factory	Fallback when Super returns nothing; also measured against Super
Nemotron 3 Nano 30B and Nemotron 3.5 Lightning	Measured for the model comparison only
Tavily	Search and extraction of a project's documentation; it retrieves, it does not generate

The demo video

Tool	What it was used for
Microsoft neural text-to-speech (via edge-tts)	The voiceover

Nothing else in the video is AI-generated: the music and sound effects were synthesised in code, the scenes are HTML animation, and the live segment is a screen recording of a real run.

Copilot, Sourcery, Cursor Bugbot, the Codex connector and the other review bots in the census were not used to build MUTINY. They appear only as subjects: their existing comments on the census pull requests were collected and checked.

Claude Code was used as a pair programmer throughout. Most of the implementation was written with it, and most of the debugging was done with it: the archive trim that silently dropped README.md and surfaced as an import error three stages later, the overlay that applied one file of a two file change and produced a TypeError no released version can raise, the address filter that missed <RetryCallState 140737342193440: ...> because it carried no at. Those were found and fixed in that loop.

The direction was not. Which problem was worth solving, what to build and what to refuse to build, which experiments would actually settle a question, what the measurements meant once they came back, and every call about what this should and should not claim: those were mine, and several of them went against what the model first suggested. The decision that the model must write inputs and never assertions came out of a measurement that said the obvious design did not work.

And none of it is taken on trust. Every number here is the output of a run that can be repeated, every finding carries the input that produced it, and the six ways this tool has been confidently wrong are written down in docs/findings-limits.md rather than quietly fixed. That standard is the whole point of the project, and it applies to how the project itself was built.