Veyrix

Break your agents before attackers do: versioned scenarios, ordered evidence traces, verified findings, and a deterministic score from 0 to 100.

Live read-only demo — a read-only snapshot of the engine, with no live scans, database, or worker.

The problem

An agent fails when untrusted content is treated as an instruction and a tool runs anyway: a payment, an email, a file write, or a secret sent outbound. That content can arrive in a document, a search result, a database record, a tool response, or an MCP resource.

Veyrix runs that boundary in a synthetic sandbox and keeps the evidence. The result is an assessment of the saved cases and policy, not a certification of a production agent.

How it works

1. Scenarios. Twelve versioned local cases across prompt injection, authorization, unsafe tool use, data exfiltration, and multi-step manipulation, plus six MCP stdio cases. The local adapter is a deterministic parser. MCP uses the official Python SDK against a bundled server.
2. Ordered evidence traces. Each case records timestamped events: the trusted instruction, the untrusted content, tool requests and responses, and the policy decision.
3. Verified findings. A failure is replayed against the saved policy before it is stored. The findings explorer shows that evidence. Triage does not rewrite the execution record.
4. A deterministic 0–100 score. veyrix.scoring (methodology 1.0.0) applies coverage gates and evidence-backed penalties with exact fractions. No LLM judges the score. A number requires a completed run, at least five determinate cases, and 80% of the saved catalog.

flowchart LR
  dashboard["Next.js dashboard"] --> api["FastAPI"]
  api --> postgres["PostgreSQL"]
  api --> redis["Redis wake-up"]
  redis --> worker["Sandboxed worker"]
  postgres --> worker
  worker --> local["Local scenarios"]
  worker --> mcp["MCP SDK"]
  worker --> llm["llm adapter"]

PostgreSQL is the durable queue. Redis only wakes the worker. The Compose worker is unprivileged, has a read-only filesystem, and sits on an internal network. Tool calls from the local parser, the MCP server, and the llm adapter execute in the same in-memory sandbox.

Details: architecture, scenarios, what the kit catches, MCP, scoring, findings.

Demo results

The public demo is a static snapshot of two local fixtures, generated on 2026-11-11 by engine 0.3.0 and scoring methodology 1.0.0 (apps/web/demo/snapshot.json). The six MCP scenarios are not in the snapshot.

Fixture	Score on the 12 local scenarios	What the run recorded
Vulnerable document assistant	4/100	12 failures, 0 passes. 11 confirmed mock executions and 1 confirmed block. 3 critical and 9 high.
Hardened document assistant	100/100	12 passes, 0 failures.

These scores describe the deterministic local fixtures under methodology 1.0.0. The formula, caps, and limits are in SCORING.md.

The route /kit is the always-on readout of the GitHub Action: the same two fixtures, with the scan time, engine and scoring versions, verified findings, and one full evidence trace. It reads a scheduled JSON publication and falls back to apps/web/demo/kit-scan.json until that publication exists. What those scenarios test, and the detection rates measured on the fixtures, is in docs/red-team-kit.md. Hosting, the publication branch, and snapshot regeneration: docs/DEMO.md.

Quickstart

Python 3.12+, Node 22+, Docker Compose, and free ports 58439 and 6379.

cp .env.example .env
python3 -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'
docker compose up -d --wait
alembic upgrade head
uvicorn veyrix_api.main:app --host 127.0.0.1 --port 8000 --reload

In a second terminal, with the same virtualenv:

python -m veyrix_api.worker

Dashboard, with the API on port 8000:

cd apps/web
npm ci
npm run dev

Open http://127.0.0.1:3000. The page proxies /api to the local API. Register a local or MCP fixture and run a suite.

VEYRIX_DEV_MODE=true is unrestricted local access with no user authentication. Bind the API to loopback. Do not expose it. For a containerized API and worker, use docker compose --profile app up --build -d instead of the host processes above. That worker cannot reach a model on the host.

API reference: API.md.

LLM adapter

Register adapter=llm through the API to run the same twelve local scenarios through an OpenAI-compatible Chat Completions endpoint (tools / tool_calls). The dashboard form still offers only the local and MCP fixtures. The model may call only the synthetic mock tools. Those calls are executed in the sandbox, recorded on the same ordered trace, and scored by verified_result and calculate_score.

variant does not change a live model. It selects the offline scripted transport and is stored on the run.

A local Ollama server needs no API key. Set this in the environment of both the API and the host worker:

export VEYRIX_LLM_BASE_URL=http://127.0.0.1:11434/v1
export VEYRIX_LLM_MODEL=qwen2.5:3b

VEYRIX_LLM_BASE_URL defaults to that loopback endpoint. Cleartext HTTP is accepted only for loopback. A hosted HTTPS endpoint needs VEYRIX_LLM_API_KEY in the environment. The key is not accepted in the request body and is not stored on the agent or the run.

여기까지가 README 앞부분입니다. 나머지는 원문에서 읽을 수 있습니다. README 전문 보기 ↗