Does your coding agent actually get better at your repository? Insi
measures it, and then works to make it so, with a person deciding what
lands.
Insi is a command-line tool (one Rust binary) you add to any git repository.
It turns the repository's own history into a benchmark, runs a coding agent
against it in throwaway checkouts, and judges every attempt with tests the
agent never saw. On top of that measurement it builds two ways for the agent
to improve:
- Lessons. After a failed attempt, the agent writes down what went wrong
and what to do instead. Later tasks are briefed with those lessons.
- The improvement loop. The agent's prompts and settings (its harness)
are changed one file at a time, and a change is kept only when the
benchmark says it is better.
Nothing changes itself silently. An improvement reaches the repository as a
draft pull request that a person reviews and merges.
Why it exists: the claim "my agent learns" is usually an anecdote. Insi's rule
is that an improvement counts only if it raises the pass rate on tasks it was
not learned from, measured against a control.
Testing and outstanding work
PR review, evidence and onboarding ·
Incord candidate execution ·
Review research and next priorities
How it works
the repository's history
│
│ insi mine every fix commit that also changed a test file becomes a task:
│ start = the commit before the fix
│ prompt = the fix commit's message
│ hidden check = the tests the fix added, kept outside the checkout
▼
tasks
│ insi validate two controls, no agent: the hidden check must FAIL at the
│ start and PASS at the fix, or the task is not a fair test
▼
fair tasks
│ insi eval the agent works each task in its own throwaway worktree;
│ the hidden tests are copied in only after it has stopped
▼
results.jsonl ──────► insi report: pass rate, timeouts, repeated failures,
│ cost, and reward hacking per condition
│
│ insi learn a failed attempt + its verified fix → one lesson
│ (the issue, the root cause, what to use, what to avoid)
▼
lessons ─────────────► briefed before later tasks, only from tasks dated earlier
│
│ insi improve change the harness one file at a time, keep only what wins
▼
draft pull request ───► a person reviews it; Insi never merges
Four rules keep the measurement honest (all eleven, with how each is enforced, are in docs/MEASUREMENT.md):
- The agent never sees the check. Hidden tests live outside the
attempt's folder and are copied in after the agent stops, replacing
anything it wrote at those paths. The attempt's repository holds the base
commit and nothing after it, so no git command shows the fix. What is on
disk (the source checkout, the hidden tests) is kept from the agent by
[worker] sandbox; insi report
counts the attempts that ran without one.
- Only fair tasks count. A task whose check already passes at the start,
or fails even with the real fix, is dropped by
insi validate.
- Learn on the past, measure on the future. Tasks are split by commit
date. Lessons are learned on the older half and measured on the newer
half, and a lesson is never briefed to the task it came from or to
anything earlier. Conditions are compared on the same task × seed pairs.
- Visible score vs real score. When a task carries a public check (the
repository's own test command), Insi records whether it passed. A public
pass with a hidden failure is counted as reward hacking: the visible score
was met without the work.
The agent is anything with a non-interactive mode: Insi runs it through a
worker command ([worker] command, or --worker "<cmd {prompt}>"). The
default is incode.
Quick start
Insi is built from source: the npm and pip packages are ready but not yet
published (RELEASING.md). You need Rust 1.89 or newer and git.
git clone https://github.com/incordai/Insi.git
cd Insi
cargo build --release # the binary is target/release/insi
In the repository you want to measure:
insi init # detect the stack, write .insi/config.toml
insi mine # tasks from the history; hidden tests go outside the checkout
insi validate # which tasks are fair tests
insi eval --fair-only # run the agent on the fair tasks
insi report # pass rate and the rest, per condition
To measure whether lessons help:
insi eval --fair-only --half older --tag phase=learn # attempts to learn from
insi learn # lessons from their failures
insi eval --fair-only --half newer --seeds 3 # lessons off
insi eval --fair-only --half newer --seeds 3 --lessons .insi-work/lessons.jsonl
insi report
This is the beginning of the README. Read the rest at the original. Read the full README ↗
