Your AI Already Has the Answer, Incord Serves It in One Call

Does your coding agent actually get better at your repository? Insi measures it, and then works to make it so, with a person deciding what lands.

Insi is a command-line tool (one Rust binary) you add to any git repository. It turns the repository's own history into a benchmark, runs a coding agent against it in throwaway checkouts, and judges every attempt with tests the agent never saw. On top of that measurement it builds two ways for the agent to improve:

• Lessons. After a failed attempt, the agent writes down what went wrong and what to do instead. Later tasks are briefed with those lessons.
• The improvement loop. The agent's prompts and settings (its harness) are changed one file at a time, and a change is kept only when the benchmark says it is better.

Nothing changes itself silently. An improvement reaches the repository as a draft pull request that a person reviews and merges.

Why it exists: the claim "my agent learns" is usually an anecdote. Insi's rule is that an improvement counts only if it raises the pass rate on tasks it was not learned from, measured against a control.

• How it works
• Quick start
• Benchmark results
• Reference: every command in detail
• Contributing · License

Testing and outstanding work

PR review, evidence and onboarding · Incord candidate execution · Review research and next priorities

How it works

 the repository's history
        │
        │  insi mine       every fix commit that also changed a test file becomes a task:
        │                    start         = the commit before the fix
        │                    prompt        = the fix commit's message
        │                    hidden check  = the tests the fix added, kept outside the checkout
        ▼
   tasks
        │  insi validate   two controls, no agent: the hidden check must FAIL at the
        │                  start and PASS at the fix, or the task is not a fair test
        ▼
   fair tasks
        │  insi eval       the agent works each task in its own throwaway worktree;
        │                  the hidden tests are copied in only after it has stopped
        ▼
   results.jsonl  ──────►  insi report: pass rate, timeouts, repeated failures,
        │                  cost, and reward hacking per condition
        │
        │  insi learn      a failed attempt + its verified fix → one lesson
        │                  (the issue, the root cause, what to use, what to avoid)
        ▼
   lessons  ─────────────► briefed before later tasks, only from tasks dated earlier
        │
        │  insi improve    change the harness one file at a time, keep only what wins
        ▼
   draft pull request ───► a person reviews it; Insi never merges

Four rules keep the measurement honest (all eleven, with how each is enforced, are in docs/MEASUREMENT.md):

1. The agent never sees the check. Hidden tests live outside the attempt's folder and are copied in after the agent stops, replacing anything it wrote at those paths. The attempt's repository holds the base commit and nothing after it, so no git command shows the fix. What is on disk (the source checkout, the hidden tests) is kept from the agent by [worker] sandbox; insi report counts the attempts that ran without one.
2. Only fair tasks count. A task whose check already passes at the start, or fails even with the real fix, is dropped by insi validate.
3. Learn on the past, measure on the future. Tasks are split by commit date. Lessons are learned on the older half and measured on the newer half, and a lesson is never briefed to the task it came from or to anything earlier. Conditions are compared on the same task × seed pairs.
4. Visible score vs real score. When a task carries a public check (the repository's own test command), Insi records whether it passed. A public pass with a hidden failure is counted as reward hacking: the visible score was met without the work.

The agent is anything with a non-interactive mode: Insi runs it through a worker command ([worker] command, or --worker "<cmd {prompt}>"). The default is incode.

Quick start

Insi is built from source: the npm and pip packages are ready but not yet published (RELEASING.md). You need Rust 1.89 or newer and git.

git clone https://github.com/incordai/Insi.git
cd Insi
cargo build --release          # the binary is target/release/insi

In the repository you want to measure:

insi init                  # detect the stack, write .insi/config.toml
insi mine                  # tasks from the history; hidden tests go outside the checkout
insi validate              # which tasks are fair tests
insi eval --fair-only      # run the agent on the fair tasks
insi report                # pass rate and the rest, per condition

To measure whether lessons help:

insi eval  --fair-only --half older --tag phase=learn    # attempts to learn from
insi learn                                               # lessons from their failures
insi eval  --fair-only --half newer --seeds 3            # lessons off
insi eval  --fair-only --half newer --seeds 3 --lessons .insi-work/lessons.jsonl
insi report

This is the beginning of the README. Read the rest at the original. Read the full README ↗