agent-roast catches the changes that turn a red build green by weakening the checks instead of fixing the code: a
loosened assertion, a skipped or deleted test, a lowered coverage threshold, a CI step allowed to fail. A linter cannot see
these, because every new line looks fine on its own. agent-roast compares the before and after of each change, as an
agent hook, a Git hook and a pull request check. It is deterministic and offline: no model, no tokens, no network.
Try it
npx agent-roast demo # ten seconds: a scripted example in a temporary repository, nothing of yours is read
npx agent-roast # review what you have not committed yet (on a clean tree: replay your last commits)
Stop your coding agent from doing it
npx agent-roast hook install # Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot and Factory Droid
For Claude Code there is also a plugin: /plugin install agent-roast --marketplace Nari-Ab/agent-roast. Any agent that speaks
MCP can also call agent-roast as a read-only tool, agent-roast mcp (setup).
When the agent is about to finish, agent-roast reviews everything it changed. If a check got weaker, the agent is sent back
once, with the file, the line and the before and after, and told to fix the code instead. If it tries again, you get a
warning rather than an endless loop. Anything agent-roast cannot check lets the agent finish: it never breaks a session. An
intended change is accepted by committing it. Not sure yet? npx agent-roast hook install --advisory only warns you and never
sends the agent back. Setup for each agent, what it does not do, and how to pause it: docs/integrations.md.
The agent hooks follow each agent's documented hook protocol and are tested with simulated hook input, not yet in long live
sessions with every agent. Reports of what you see are welcome.
Check every pull request
# .github/workflows/agent-roast.yml
name: Agent Roast
on: pull_request
permissions: { contents: read, pull-requests: write }
jobs:
audit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with: { fetch-depth: 0 }
- uses: Nari-Ab/agent-roast@v1
with: { comment: true } # one comment, updated on every push; add `advisory: true` to try it without failing PRs
Also: npx agent-roast review --base origin/main in any CI, npx agent-roast init --hook or the
pre-commit framework for the next commit, and SARIF output for code scanning.
Policy is read from the base branch, so a pull request cannot switch off the rules that review it. Before you decide,
npx agent-roast replay shows what it would have said about your last 50 commits.
What it catches
| Rule | Fires when | By default |
|---|
assertion-weakened | an exact check is replaced by a weaker one: toBe(42) → toBeDefined(), toBe(2) → toBeLessThanOrEqual(2), toBe(2) → not.toBe(0), toEqual(x) → toMatchObject(x), assert x == 1 → assert x | blocks |
guard-weakened | a coverage threshold is lowered, strict is switched off, a lint rule is downgraded, a CI step may fail, a workflow stops running tests, a conftest.py hook sets a report to passed | blocks |
test-skip | a test that was already there is switched off or expected to fail: it.skip, @pytest.mark.skip, @pytest.mark.skipif(True), @pytest.mark.xfail, it.fails, test.fixme, t.Skip, #[ignore], @Disabled, [Ignore], xit, ... (a new test that is skipped is not reported) | blocks |
test-focus, trivial-assert | .only / fit; expect(true).toBe(true) and its relatives in ten languages, assert x or True; a test that asserts on a function it mocked itself | blocks |
assertion-removed, test-removed, expectation-rewritten | a test loses assertions, disappears, or its expected value is edited next to production code | warns |
swallowed-error, type-escape, stub-abandonment | catch {} / except: pass, as any / # type: ignore / # pragma: no cover / istanbul ignore, throw new Error("Not implemented") | warns |
Renames, moved tests, parametrization, moved assertions and reformatting are recognized and not reported, and neither is a skip
that depends on the environment (if testing.Short() { t.Skip() }, test.skip(isMobile, ...)). JavaScript/TypeScript, Python
and Go get the full before/after comparison; skips and placeholder assertions also cover Rust, Java, Kotlin, C#, Ruby, PHP and
Swift; CI and config files are checked whatever the language. Each rule with examples and limits: docs/rules.md.
Configure
Rules, severities and paths live in .agentroastrc.json at the repository root (npx agent-roast init writes a starting file):
{
"failOn": ["assertion-weakened", "guard-weakened", "test-skip", "test-focus", "trivial-assert"],
"rules": { "type-escape": "off" },
"ignorePaths": ["**/generated/**"]
}
A change cannot edit this file to relax its own review, because it is read from HEAD or the base branch. To accept one
intentional finding, add an expiring exception instead of turning a rule off, and adopt an existing project without blocking on
old debt with a baseline: docs/reference.md.
Does it work?
Measured, with the denominators and the caveats in docs/benchmarks.md. Weakened checks are rare, mostly
deliberate, and not specific to AI. agent-roast is a tripwire that asks about them. On repositories it had not seen, between a quarter and
two thirds of its blocks were questions with a harmless answer, depending on the sample.
This is the beginning of the README. Read the rest at the original. Read the full README ↗
