Steadygate

A statistical release gate for non-deterministic systems: LLM agents, evals, and flaky test suites.

▶ Try the live demo, which runs in your browser with nothing to install.

Your eval passed, and your agent still got worse. Gates like "fail if score < 0.80" break down when the model, the LLM judge, and the set of cases that ran all vary between runs. They raise false alarms on noise, and they pass quietly when half the suite crashed.

Steadygate sits on top of the eval or test framework you already use. It compares each run against your own history and gives one of three answers:

Exit	Verdict	Meaning
0	PASS	Within normal variation of the baseline
1	REGRESSION	A required rule failed: this is worse, not noise
2	INCONCLUSIVE	Results errored or were missing, so the gate refuses to call it a pass
3	config error	Bad config or unreadable results file

The core rule is that errors can never make you pass. A crashed test or a judge timeout is reported as inconclusive, never as a silent green.

Status: alpha (v0.1.0). The config format may still change before 1.0; the exit codes won't.

Who it's for

• AI and LLM app teams. You run evals with promptfoo, DeepEval or your own script, and a fixed threshold either blocks good releases or lets bad ones through. You get a go/no-go that knows your normal variation.
• QA and release owners. You sign off releases, and now you're asked to test AI features whose outputs vary. You get one clear verdict per run, an exit code your pipeline can act on, and a report in the pull request.
• Teams with flaky tests. Your noisy end-to-end, performance or game tests cry wolf. You get repeated-trial pass rates with confidence intervals, so 7/10 isn't treated as a failure, and crashes are kept apart from real regressions.

If your tests give the same answer every time, a plain pass/fail is all you need.

Quick start

Download the installer for your system:

• Windows
• Mac (Apple silicon). On an Intel Mac, install with pip instead.
• Linux, for x86_64 (64-bit Intel/AMD) (run chmod +x steadygate-setup-linux first)

The builds are not signed yet. Windows may say "Windows protected your PC": choose More info, then Run anyway. Mac: Unzip it, then open the app. If macOS says it can't be checked, go to System Settings → Privacy & Security and choose Open Anyway.

Then, in your project folder, set it up:

steadygate init

steadygate init writes a steadygate.yml and a GitHub workflow for you. Use --dir to pick the folder, --format (promptfoo, deepeval, pytest-json, playwright, junit or jsonl), --results, --metric, --tolerance and --eval-command to fill in your details, and --force to overwrite files that already exist.

Or install from PyPI (Python 3.11 or newer):

pipx install steadygate        # or: pip install steadygate

To write the config by hand, make a steadygate.yml:

store: .steadygate/history.db
window: 10          # compare against the last 10 recorded runs
min_history: 3      # don't gate until 3 runs exist

rules:
  - type: relative_delta          # mean may not drop >5% below the baseline median
    metric: accuracy
    max_regression: 0.05

  - type: absolute                # hard floor on every case
    metric: accuracy
    min: 0.5
    scope: case

  - type: relative_delta          # tracked, but doesn't block
    metric: latency_ms
    direction: lower_is_better
    max_regression: 0.25
    required: false

Record baselines on your main branch and gate on pull requests:

steadygate record results.jsonl --run-id "$GITHUB_SHA"     # on main
steadygate run results.jsonl --markdown report.md          # on PRs

steadygate run --record gates first and adds the run to the baseline only if it passes.

Try the example

cd examples/basic
for i in 1 2 3; do steadygate record baseline-$i.jsonl --run-id base-$i; done
steadygate run candidate-noise.jsonl        # PASS: -2.2% is within tolerance
steadygate run candidate-regression.jsonl   # REGRESSION: -17.8%
steadygate run candidate-broken.jsonl       # INCONCLUSIVE: one case's judge timed out
steadygate run candidate-missing.jsonl      # INCONCLUSIVE: one case never reported at all

cd ../trials                                # each case run 10 times
steadygate run candidate-noise.jsonl        # PASS: 7/10 against an 80% target is noise
steadygate run candidate-broken.jsonl       # REGRESSION: 3/10 is confidently below 80%

Or try these scenarios in the live demo, where you can move the scores yourself.

Find flaky cases

steadygate flaky reads one or more results files (oldest first) and scores every case on its pass/fail trials. Repeated runs inside one file count too, such as Playwright's --repeat-each or an eval run 5 times.

steadygate flaky nightly-1.xml nightly-2.xml nightly-3.xml
steadygate flaky --history                 # the runs recorded in the baseline store
steadygate flaky results.json --markdown flaky.md --json

steadygate flaky: 2 flaky, 1 passing (metric: passed)
  case          status  passed  pass rate      flips
  cancel-order  flaky   10/20   50% (30%-70%)  3 (16%)
  book-flight   flaky   18/20   90% (70%-97%)  3 (16%)

Quarantine candidates:
  - cancel-order
  - book-flight

以上为 README 的开头部分。其余内容请阅读原文。 查看完整 README ↗