ACE Monorepo
Agent Certification Engine
Table of Contents
About ACE
ACE stands for Agent Certification Engine — a platform for evaluating, fault-injecting, and certifying autonomous agents under controlled chaos conditions. This monorepo aggregates every component required to run the platform end-to-end.
- Purpose. Produce reproducible certification reports that quantify how an agent behaves under faults, with evidence (traces, metrics, ground truth) preserved at every phase.
- Architecture. A GraphQL control plane (AgentCert) drives chaos experiments on Kubernetes; the certifier consumes the resulting Langfuse traces and emits a multi-phase certification artifact (JSON + PDF).
- Pipeline. Phase 0 (trace ingest) → Phase 1 (fault bucketing) → Phase 2 (statistical aggregation) → Phase 3 (certificate build).
- Local stack. MongoDB for persistence, Langfuse for OTEL trace storage, LiteLLM as the unified LLM gateway in front of Azure OpenAI.
- Layout. Each concern lives in its own submodule (see Submodules) so components can be versioned, built, and released independently.
Pipeline in action
Click the GIF to open the interactive version on the docs site.
Documentation
All project documentation lives in docs/:
New to the platform? Go straight to docs/setup/ and the Quick Start below.
Submodules
| Module | Description |
|---|
| AgentCert | Core AgentCert platform — GraphQL control plane, auth, web UI |
| app-charts | Application Helm charts (e.g. sock-shop) + the install-app image |
| agent-charts | Agent Helm charts + the install-agent image |
| certifier | Certification engine (FastAPI pipeline → JSON + PDF report) |
| flash-agent | Flash agent implementation (the agent under test) |
| agent-sidecar | Sidecar that proxies the agent's LLM/MCP traffic |
| agentcert-stack | Full-stack deployment assets (LiteLLM setup, etc.) |
| chaos-charts | Chaos experiment & fault definitions (tracks master) |
| litmus-go | Litmus chaos experiment executors (Go) |
Agents
Agents in agents/
agents/ collects every agent implementation considered or exercised while certifying an
open-weight model against ACE's chaos/compliance scenarios — one place to find "what
agent, from where, running on what framework."
| Agent | Location | Framework | Role |
|---|
| flash-agent | agents/flash-agent/ (inlined; the former standalone flash-agent/ submodule at the repo root has been removed) | Custom ReAct loop (OpenAI-compatible SDK + MCP tools) | The primary agent under test for SRE fault scenarios. Discovers Kubernetes/Prometheus MCP tools at startup, reasons over multiple tool-calling iterations, and emits a structured JSON health analysis with remediation guidance. No native tracing of its own — relies on the LiteLLM gateway's Langfuse callback. |
| CISO Agent | agents/ciso-agent/ (inlined from upstream itbench-hub/ITBench-CISO-CAA-Agent, with fixes) | CrewAI + LangGraph | ITBench's reference CISO agent. Generates and deploys compliance policies (Kyverno ClusterPolicy, OPA Rego) against injected CIS Benchmark violations, then reports the deployed resource. Exercised end-to-end against a real Gen-CIS-b-K8s-Kyverno scenario — see OPEN_WEIGHT_CERTIFICATION_HANDOFF.md §8 for the full trial and the two bugs found in its LLM integration. |
| SRE Agent ("Zero") | agents/sre-agent/ (inlined from upstream itbench-hub/ITBench-CISO-SRE-FinOps-Agent, unmodified) | Wrapper around OpenAI's Codex CLI | ITBench's other reference agent (despite the repo name, covers SRE/CISO/FinOps scenarios). Considered as a candidate for the SRE certification track but not the one actually exercised — flash-agent was used instead, and the CISO Agent above was used for the CISO track per explicit instruction to use "the CrewAI based agent." |
Using CrewAI with a single agent
The CISO Agent trial is the reference example in this repo for pointing CrewAI at a
local, OpenAI-compatible endpoint (e.g. a self-hosted open-weight model behind Ollama or
a LiteLLM proxy) instead of a hosted API. The pattern, extracted from
agents/ciso-agent/src/ciso_agent/llm.py:
from crewai import Agent, Crew, Process, Task, LLM
llm = LLM(
model="openai/qwen2.5-7b-instruct", # see the gotcha below
base_url="http://127.0.0.1:14000/v1", # your OpenAI-compatible endpoint
api_key="sk-...", # any non-empty string if the endpoint doesn't check it
temperature=0.1,
)
agent = Agent(role="Test", goal="...", backstory="", llm=llm, verbose=True)
task = Task(name="target_task", description="...", expected_output="...", agent=agent, tools=[...])
crew = Crew(name="MyCrew", agents=[agent], tasks=[task], process=Process.sequential, verbose=True)
result = crew.kickoff(inputs={})
The gotcha: CrewAI's LLM class calls litellm.completion() directly, which requires
an explicit provider prefix (openai/<model>) to resolve a provider via
get_llm_provider() — even with base_url set. A bare model name (e.g. just
qwen2.5-7b-instruct, matching your proxy's own model alias) crashes with
litellm.BadRequestError: LLM Provider NOT provided. This is easy to get backwards: if
your endpoint is a raw LangChain ChatOpenAI call instead (a different code path,
sometimes used for auxiliary/non-Crew LLM calls in the same codebase), the opposite is
true — it sends model as-is to base_url, so the same openai/ prefix would break
that path instead. One model-string value has to work for both if your code mixes the
two; see agents/ciso-agent/src/ciso_agent/llm.py's init_agent_llm() vs. init_llm()
for how this repo resolves it (prefix only added when the model string doesn't already
declare a provider).
Verified end-to-end with qwen2.5:7b-instruct served locally via Ollama, routed through
the LiteLLM gateway described in Submodules — see
OPEN_WEIGHT_CERTIFICATION_HANDOFF.md §8.3 for
both real bugs found getting this working and how they were fixed.
Getting the Code
Clone with submodules
git clone --recurse-submodules <repo-url>
Initialize submodules (if already cloned)
git submodule update --init --recursive
./scripts/setup.sh --deploy also detects an ordinary clone with missing submodules and
initializes them before it builds anything, so this is safe to omit when you use
the setup wizard.
Update all submodules to latest
git submodule update --remote --merge
Quick Start (Kubernetes)
Everything — MongoDB, the AgentCert control plane (auth + GraphQL + UI), LiteLLM,
Langfuse, and the Certifier — runs inside a local kind
Kubernetes cluster managed by scripts/setup.sh:
git clone --recurse-submodules <repo-url>
cd ace-monorepo
./scripts/setup.sh --deploy # wizard: creates .env, creates kind cluster, deploys to K8s
--deploy is required. ./scripts/setup.sh with no flags builds every image
and exits — it writes no .env, creates no cluster and asks nothing. It ends with
All images built successfully! followed by Next: ./scripts/setup.sh --deploy.
That is a successful build, not an install.
The wizard asks for your LLM credentials; press Enter at every other prompt. The defaults
build every first-party image and the immutable hub bundle (app/agent/fault catalogs + charts)
from this checkout, deploy with Helm into a per-user kind cluster, register and connect the
ace-local chaos infrastructure, and seed the demo experiments — ready to run an experiment
with no further manual steps. Setup stops with an explicit error rather than deploying a stack
that would fail later (missing image, unbuildable hub bundle).
📚 Detailed setup guides — docs/setup/
1. Prerequisites
| Tool | Min version | Install |
|---|
| Docker (+ Compose v2) | 28+ | sudo apt-get install docker.io — user must be in docker group |
| kind | v0.20+ | kind releases |
| kubectl | v1.27+ | sudo snap install kubectl --classic or kubectl install |
| helm | v3.12+ | helm install |
| git, python3 | any / 3.8+ | sudo apt-get install git python3 |
./scripts/setup.sh --deploy checks all of them first (scripts/check-prerequisites.sh).
2. Run the setup wizard
./scripts/setup.sh --deploy # first time (interactive)
./scripts/setup.sh --restart # later: re-apply saved answers, no prompts
It creates .env from .env.example, picks collision-free host ports and a per-user
instance name, then — with the default answers — builds everything locally, creates the
kind cluster agentcert-<ACE_INSTANCE_NAME>, deploys via Helm (k at the deploy prompt
selects plain kubectl apply of deploy/k8s/ instead), connects the chaos infrastructure
and prints the URLs. See docs/setup/route-2-fresh-kind.md
for the step-by-step flow and docs/setup/running-an-experiment.md
for your first experiment.
3. Services
Reachable at
4. Day-to-day operations
kubectl get pods -n ace # health check
kubectl logs -n ace deploy/graphql -f # tail the control plane
kubectl rollout restart -n ace deploy/graphql # restart one service
./scripts/setup.sh --restart # re-apply after manifest/.env changes
./scripts/shut_down.sh # tear down this checkout (dumps MongoDB first)
Ports in the table above are defaults; on a shared host setup.sh may pick others — the
actual URLs are printed at the end of setup and saved in .env (KIND_HOSTPORT_*).
Local Development (host processes)
For developing the Go backend or React frontend with hot-reload, the original
host-process flow is fully documented at
docs/setup/local-dev.md. Quick reference:
cp .env.example .env
cp build-paths.env.example build-paths.env
./scripts/start-local-services.sh --deploy # MongoDB + Langfuse + LiteLLM + Certifier (Docker)
bash scripts/azure_build/start-agentcert-v2.sh --deploy \
--env-file $(pwd)/.env \
--paths-file $(pwd)/build-paths.env # auth + GraphQL + frontend on the host
Certifier API service (Dockerized)
The full FastAPI certifier — Phase 0 (fault bucketing) → Phase 1 (metrics extraction) → Phase 2 (aggregation) → Phase 3 (12-section certification report) → HTML + PDF rendering via playwright — runs as the certifier deployment in the ace namespace and is deployed automatically by scripts/setup.sh at http://localhost:18000/docs. It shares the monorepo's MongoDB rather than shipping its own.
The certifier image is published at docker.io/agentcert/certifier:latest and pulled automatically by Kubernetes (imagePullPolicy: IfNotPresent).
To restart or check the certifier in Kubernetes:
kubectl get pods -n ace -l app=certifier
kubectl logs -n ace deploy/certifier -f
kubectl rollout restart -n ace deploy/certifier
All credentials are injected from the ace-env Secret (created from .env by scripts/setup.sh). On a new host: clone, run ./scripts/setup.sh --deploy and press Enter at the Choice [k/H/n] deploy prompt — the certifier comes up alongside every other service.
Quick command reference
Once kubectl get pods -n ace -l app=certifier shows Running, open
http://localhost:18000/docs. The endpoints are:
| Method | Path | What it does |
|---|
POST | /api/v1/bucketing-extraction | Phase 0+1: fetch a trace (Langfuse or file), classify events into per-fault buckets, extract per-fault metrics. Returns task_id. |
POST | /api/v1/aggregation-certification | Phase 2+3: aggregate every *_metrics.json under the experiment's fault-bucketing tree, build a 12-section certification report, render HTML + PDF. Returns cert_task_id. |
GET | /api/v1/tasks | Poll a bucketing/extraction task by experiment_id + experiment_run_id. |
GET | /api/v1/cert-tasks | Poll a certification task by experiment_id. |
Outputs land under certifier/workspace/{agent_id}/{experiment_id}/:
fault-bucketing/{run_id}/
traces/raw_trace.json
fault_buckets/{raw_trace_bucket_*.json, bucketing_manifest.json, batch_classification_trace.json}
ground_truth/*.json
metrics/*_metrics.json ← input to Phase 2
pipeline_summary.json
aggregation/aggregation.json
cert-builder/certification.json
certification/
cert-{agent_id}-YYYY-MM-DD.html
cert-{agent_id}-YYYY-MM-DD.pdf
End-to-end smoke test (one Langfuse run, ~10 min wall-clock):
AGENT="<agent_id>"; EXP="<experiment_id>"; RID="<experiment_run_id>"
# Phase 0+1
curl -s -X POST -H "Content-Type: application/json" -d "$(cat <<EOF
{"agent_id":"${AGENT}","experiment_id":"${EXP}","run_id":"${RID}",
"trace_source":{"type":"langfuse"},"storage_config":{"type":"local"}}
EOF
)" http://localhost:18000/api/v1/bucketing-extraction
# Poll
curl -s "http://localhost:18000/api/v1/tasks?experiment_id=${EXP}&experiment_run_id=${RID}"
# Phase 2+3 (consumes every metrics.json under the experiment)
curl -s -X POST -H "Content-Type: application/json" -d "$(cat <<EOF
{"agent_id":"${AGENT}","agent_name":"vaya","experiment_id":"${EXP}",
"runs_per_fault":5,"storage_config":{"type":"local"}}
EOF
)" http://localhost:18000/api/v1/aggregation-certification
# Poll
curl -s "http://localhost:18000/api/v1/cert-tasks?experiment_id=${EXP}"
Certifier Dev Tools
Two helper scripts in scripts/ run the certifier pipeline locally without starting the FastAPI service. Both invoke the same TraceService / BucketPipelineService / CertPipelineService code paths used by the running service, and read LANGFUSE_* / AZURE_OPENAI_* from the root .env.
Dump a Langfuse trace into pipeline-compatible JSON
scripts/dump_langfuse_trace.py fetches a single trace from Langfuse and writes raw_trace.json plus a trace_meta.json summary. This is useful for collecting offline samples that the certifier can later consume via a FileTraceSource.
./scripts/dump_langfuse_trace.py \
--experiment-id <UUID> \
--run-id <UUID>
# or override the dump dir
./scripts/dump_langfuse_trace.py \
--experiment-id <UUID> --run-id <UUID> \
--output-dir ./.tmp/sample-001
Run the full certification end-to-end
scripts/run_certification.py runs Phase 0+1+2+3 against one trace and writes outputs into .tmp/ (gitignored) using the same on-disk layout the production API workers create:
.tmp/{agent_id}/{experiment_id}/
fault-bucketing/{run_id}/
traces/raw_trace.json
fault_buckets/*.json
metrics/*_metrics.json
ground_truth/*.json
pipeline_summary.json
aggregation/aggregation.json
cert-builder/certification.json ← final certificate (JSON)
cert-builder/certification.pdf ← rendered PDF (auto-generated)
# Look up agent/experiment/run IDs automatically from a Langfuse trace ID
./scripts/run_certification.py --trace-id <LANGFUSE_TRACE_UUID>
# Or pass the IDs directly (no metadata lookup)
./scripts/run_certification.py \
--agent-id <UUID> \
--experiment-id <UUID> \
--run-id <UUID> \
--agent-name vaya
# Stop after Phase 0+1 (skip aggregation + cert builder)
./scripts/run_certification.py --trace-id <UUID> --skip-cert
Optional flags: --workspace <dir> (default .tmp/), --batch-size (LLM classification batch size, default 10), --runs-per-fault (default 1, used for Phase 2 statistical-significance checks), --no-pdf (skip the PDF render step), and --debug (retain Phase 3 intermediates).
Render a certification report to PDF
The run_certification.py script renders the PDF automatically. To re-render an existing certification.json (or one produced by the running service), use scripts/render_certification_pdf.py:
pip install --user reportlab # one-time, only needed for PDF output
./scripts/render_certification_pdf.py \
--input .tmp/<agent_id>/<exp_id>/cert-builder/certification.json
# → writes certification.pdf next to the input
# or pick a custom output path
./scripts/render_certification_pdf.py \
--input certification.json \
--output ./out/agent-cert.pdf
Admin Actions
Build Docker images locally
Build every project-owned image in deploy/images.txt:
make build
# equivalent: ./scripts/build-and-push.sh
# equivalent: ./scripts/setup.sh ← builds and EXITS; it does not install anything
These defaults build locally without login, push, image pruning, cluster loading,
or deployment. The run ends with:
======================================
All images built successfully!
======================================
Build-only run: nothing was deployed and no cluster was changed.
Next: ./scripts/setup.sh --deploy (configure .env, create the cluster and install ACE)
For a faster rebuild, use make build BUILD_ARGS="--allow-build-cache".
Publish explicitly with ./scripts/build-and-push.sh --push --tag RELEASE-N
using a new release tag, or install explicitly with ./scripts/setup.sh --deploy.
--kind-load explicitly loads locally built images into KinD.
Component build wrappers use the same options and local-only default.
See the script mode guide for every entry point and validation.
Community & Contributing
Questions, ideas, or want to help build ACE? Join us on Slack:
👉 AgentCert Slack workspace
See CONTRIBUTING.md for setup, branching, commit, issue, and pull-request guidelines.
License
Released under the Apache License 2.0.
