ACE Monorepo

Agent Certification Engine

---

Table of Contents

• About ACE
• Documentation
• Submodules
• Agents
  • Agents in agents/
  • Using CrewAI with a single agent
• Getting the Code
• Quick Start (Kubernetes)
  • 1. Prerequisites
  • 2. Run the setup wizard
  • 3. Services
  • 4. Day-to-day operations
  • 📚 Detailed setup guides (docs/setup/)
• Local Development (host processes)
• Certifier API service (Dockerized)
• Certifier Dev Tools
  • Dump a Langfuse trace into pipeline-compatible JSON
  • Run the full certification end-to-end
  • Render a certification report to PDF
• Admin Actions
  • Build Docker images locally
• Community & Contributing
• License

---

About ACE

ACE stands for Agent Certification Engine — a platform for evaluating, fault-injecting, and certifying autonomous agents under controlled chaos conditions. This monorepo aggregates every component required to run the platform end-to-end.

• Purpose. Produce reproducible certification reports that quantify how an agent behaves under faults, with evidence (traces, metrics, ground truth) preserved at every phase.
• Architecture. A GraphQL control plane (AgentCert) drives chaos experiments on Kubernetes; the certifier consumes the resulting Langfuse traces and emits a multi-phase certification artifact (JSON + PDF).
• Pipeline. Phase 0 (trace ingest) → Phase 1 (fault bucketing) → Phase 2 (statistical aggregation) → Phase 3 (certificate build).
• Local stack. MongoDB for persistence, Langfuse for OTEL trace storage, LiteLLM as the unified LLM gateway in front of Azure OpenAI.
• Layout. Each concern lives in its own submodule (see Submodules) so components can be versioned, built, and released independently.

---

Pipeline in action

Click the GIF to open the interactive version on the docs site.

---

Documentation

All project documentation lives in docs/:

Area	Where	What's inside
Setup & operations	docs/setup/	One-command bring-up, the three cluster routes, configuration & port reference, and the full experiment walkthrough. Start here to run the platform.
Architecture	docs/architecture.md	End-to-end system architecture of the control plane + certifier.
Workflow animation	docs/animated_archiecture/	Self-contained HTML animation of the AgentCert workflow (open in a browser).
Methodology	docs/Methodologies/	The certification methodology: introduction, experiment design, metrics, pipeline, certification, observations.
API

New to the platform? Go straight to docs/setup/ and the Quick Start below.

---

Submodules

Module	Description
AgentCert	Core AgentCert platform — GraphQL control plane, auth, web UI
app-charts	Application Helm charts (e.g. sock-shop) + the install-app image
agent-charts	Agent Helm charts + the install-agent image
certifier	Certification engine (FastAPI pipeline → JSON + PDF report)
flash-agent	Flash agent implementation (the agent under test)
agent-sidecar	Sidecar that proxies the agent's LLM/MCP traffic
agentcert-stack	Full-stack deployment assets (LiteLLM setup, etc.)
chaos-charts	Chaos experiment & fault definitions (tracks master)
litmus-go	Litmus chaos experiment executors (Go)

---

Agents

Agents in agents/

agents/ collects every agent implementation considered or exercised while certifying an open-weight model against ACE's chaos/compliance scenarios — one place to find "what agent, from where, running on what framework."

Agent	Location	Framework	Role
flash-agent	agents/flash-agent/ (inlined; the former standalone flash-agent/ submodule at the repo root has been removed)	Custom ReAct loop (OpenAI-compatible SDK + MCP tools)	The primary agent under test for SRE fault scenarios. Discovers Kubernetes/Prometheus MCP tools at startup, reasons over multiple tool-calling iterations, and emits a structured JSON health analysis with remediation guidance. No native tracing of its own — relies on the LiteLLM gateway's Langfuse callback.
CISO Agent	agents/ciso-agent/ (inlined from upstream itbench-hub/ITBench-CISO-CAA-Agent, with fixes)	CrewAI + LangGraph	ITBench's reference CISO agent. Generates and deploys compliance policies (Kyverno ClusterPolicy, OPA Rego) against injected CIS Benchmark violations, then reports the deployed resource. Exercised end-to-end against a real Gen-CIS-b-K8s-Kyverno scenario — see OPEN_WEIGHT_CERTIFICATION_HANDOFF.md §8 for the full trial and the two bugs found in its LLM integration.
SRE Agent ("Zero")	agents/sre-agent/ (inlined from upstream itbench-hub/ITBench-CISO-SRE-FinOps-Agent, unmodified)	Wrapper around OpenAI's Codex CLI	ITBench's other reference agent (despite the repo name, covers SRE/CISO/FinOps scenarios). Considered as a candidate for the SRE certification track but not the one actually exercised — flash-agent was used instead, and the CISO Agent above was used for the CISO track per explicit instruction to use "the CrewAI based agent."

Using CrewAI with a single agent

The CISO Agent trial is the reference example in this repo for pointing CrewAI at a local, OpenAI-compatible endpoint (e.g. a self-hosted open-weight model behind Ollama or a LiteLLM proxy) instead of a hosted API. The pattern, extracted from agents/ciso-agent/src/ciso_agent/llm.py:

from crewai import Agent, Crew, Process, Task, LLM

llm = LLM(
    model="openai/qwen2.5-7b-instruct",   # see the gotcha below
    base_url="http://127.0.0.1:14000/v1", # your OpenAI-compatible endpoint
    api_key="sk-...",                     # any non-empty string if the endpoint doesn't check it
    temperature=0.1,
)

agent = Agent(role="Test", goal="...", backstory="", llm=llm, verbose=True)
task = Task(name="target_task", description="...", expected_output="...", agent=agent, tools=[...])
crew = Crew(name="MyCrew", agents=[agent], tasks=[task], process=Process.sequential, verbose=True)

result = crew.kickoff(inputs={})

The gotcha: CrewAI's LLM class calls litellm.completion() directly, which requires an explicit provider prefix (openai/<model>) to resolve a provider via get_llm_provider() — even with base_url set. A bare model name (e.g. just qwen2.5-7b-instruct, matching your proxy's own model alias) crashes with litellm.BadRequestError: LLM Provider NOT provided. This is easy to get backwards: if your endpoint is a raw LangChain ChatOpenAI call instead (a different code path, sometimes used for auxiliary/non-Crew LLM calls in the same codebase), the opposite is true — it sends model as-is to base_url, so the same openai/ prefix would break that path instead. One model-string value has to work for both if your code mixes the two; see agents/ciso-agent/src/ciso_agent/llm.py's init_agent_llm() vs. init_llm() for how this repo resolves it (prefix only added when the model string doesn't already declare a provider).

Verified end-to-end with qwen2.5:7b-instruct served locally via Ollama, routed through the LiteLLM gateway described in Submodules — see OPEN_WEIGHT_CERTIFICATION_HANDOFF.md §8.3 for both real bugs found getting this working and how they were fixed.

---

Getting the Code

Clone with submodules

git clone --recurse-submodules <repo-url>

Initialize submodules (if already cloned)

git submodule update --init --recursive

./scripts/setup.sh --deploy also detects an ordinary clone with missing submodules and initializes them before it builds anything, so this is safe to omit when you use the setup wizard.

Update all submodules to latest

git submodule update --remote --merge

---

Quick Start (Kubernetes)

Everything — MongoDB, the AgentCert control plane (auth + GraphQL + UI), LiteLLM, Langfuse, and the Certifier — runs inside a local kind Kubernetes cluster managed by scripts/setup.sh:

git clone --recurse-submodules <repo-url>
cd ace-monorepo
./scripts/setup.sh --deploy          # wizard: creates .env, creates kind cluster, deploys to K8s

--deploy is required. ./scripts/setup.sh with no flags builds every image and exits — it writes no .env, creates no cluster and asks nothing. It ends with All images built successfully! followed by Next: ./scripts/setup.sh --deploy. That is a successful build, not an install.

The wizard asks for your LLM credentials; press Enter at every other prompt. The defaults build every first-party image and the immutable hub bundle (app/agent/fault catalogs + charts) from this checkout, deploy with Helm into a per-user kind cluster, register and connect the ace-local chaos infrastructure, and seed the demo experiments — ready to run an experiment with no further manual steps. Setup stops with an explicit error rather than deploying a stack that would fail later (missing image, unbuildable hub bundle).

📚 Detailed setup guides — docs/setup/

• Setup overview & prerequisites
• Configuration & changing ports — every .env switch + how to move busy ports
• Route 2 — fresh VM with kind
• Route 3 — cloud Kubernetes (AKS/EKS/GKE)
• Local development (host processes) — Go/Node on the host, Docker for infra
• Running your first experiment — create → run → certify (setup already connected the chaos infrastructure)

1. Prerequisites

Tool	Min version	Install
Docker (+ Compose v2)	28+	sudo apt-get install docker.io — user must be in docker group
kind	v0.20+	kind releases
kubectl	v1.27+	sudo snap install kubectl --classic or kubectl install
helm	v3.12+	helm install
git, python3	any / 3.8+	sudo apt-get install git python3

./scripts/setup.sh --deploy checks all of them first (scripts/check-prerequisites.sh).

2. Run the setup wizard

./scripts/setup.sh --deploy              # first time (interactive)
./scripts/setup.sh --restart    # later: re-apply saved answers, no prompts

It creates .env from .env.example, picks collision-free host ports and a per-user instance name, then — with the default answers — builds everything locally, creates the kind cluster agentcert-<ACE_INSTANCE_NAME>, deploys via Helm (k at the deploy prompt selects plain kubectl apply of deploy/k8s/ instead), connects the chaos infrastructure and prints the URLs. See docs/setup/route-2-fresh-kind.md for the step-by-step flow and docs/setup/running-an-experiment.md for your first experiment.

3. Services

Reachable at

Service	URL	Default login
AgentCert UI	http://localhost:2001	admin / litmus
GraphQL (REST/WS)	http://localhost:8081	—
Auth (REST)	http://localhost:3005	—
Certifier (Swagger)	http://localhost:18000/docs	—
LiteLLM proxy	http://localhost:14000	—
Langfuse	http://localhost:4002	[email protected] / agentcert-admin
MongoDB	mongodb://admin:1234@localhost:27017/?replicaSet=rs0&authSource=admin	—

4. Day-to-day operations

kubectl get pods -n ace                         # health check
kubectl logs -n ace deploy/graphql -f           # tail the control plane
kubectl rollout restart -n ace deploy/graphql   # restart one service
./scripts/setup.sh --restart                    # re-apply after manifest/.env changes
./scripts/shut_down.sh                          # tear down this checkout (dumps MongoDB first)

Ports in the table above are defaults; on a shared host setup.sh may pick others — the actual URLs are printed at the end of setup and saved in .env (KIND_HOSTPORT_*).

---

Local Development (host processes)

For developing the Go backend or React frontend with hot-reload, the original host-process flow is fully documented at docs/setup/local-dev.md. Quick reference:

cp .env.example .env
cp build-paths.env.example build-paths.env
./scripts/start-local-services.sh --deploy          # MongoDB + Langfuse + LiteLLM + Certifier (Docker)
bash scripts/azure_build/start-agentcert-v2.sh --deploy \
    --env-file   $(pwd)/.env \
    --paths-file $(pwd)/build-paths.env     # auth + GraphQL + frontend on the host

Certifier API service (Dockerized)

The full FastAPI certifier — Phase 0 (fault bucketing) → Phase 1 (metrics extraction) → Phase 2 (aggregation) → Phase 3 (12-section certification report) → HTML + PDF rendering via playwright — runs as the certifier deployment in the ace namespace and is deployed automatically by scripts/setup.sh at http://localhost:18000/docs. It shares the monorepo's MongoDB rather than shipping its own.

The certifier image is published at docker.io/agentcert/certifier:latest and pulled automatically by Kubernetes (imagePullPolicy: IfNotPresent).

To restart or check the certifier in Kubernetes:

kubectl get pods -n ace -l app=certifier
kubectl logs -n ace deploy/certifier -f
kubectl rollout restart -n ace deploy/certifier

All credentials are injected from the ace-env Secret (created from .env by scripts/setup.sh). On a new host: clone, run ./scripts/setup.sh --deploy and press Enter at the Choice [k/H/n] deploy prompt — the certifier comes up alongside every other service.

Quick command reference

Once kubectl get pods -n ace -l app=certifier shows Running, open http://localhost:18000/docs. The endpoints are:

Method	Path	What it does
POST	/api/v1/bucketing-extraction	Phase 0+1: fetch a trace (Langfuse or file), classify events into per-fault buckets, extract per-fault metrics. Returns task_id.
POST	/api/v1/aggregation-certification	Phase 2+3: aggregate every *_metrics.json under the experiment's fault-bucketing tree, build a 12-section certification report, render HTML + PDF. Returns cert_task_id.
GET	/api/v1/tasks	Poll a bucketing/extraction task by experiment_id + experiment_run_id.
GET	/api/v1/cert-tasks	Poll a certification task by experiment_id.

Outputs land under certifier/workspace/{agent_id}/{experiment_id}/:

fault-bucketing/{run_id}/
  traces/raw_trace.json
  fault_buckets/{raw_trace_bucket_*.json, bucketing_manifest.json, batch_classification_trace.json}
  ground_truth/*.json
  metrics/*_metrics.json          ← input to Phase 2
  pipeline_summary.json
aggregation/aggregation.json
cert-builder/certification.json
certification/
  cert-{agent_id}-YYYY-MM-DD.html
  cert-{agent_id}-YYYY-MM-DD.pdf

End-to-end smoke test (one Langfuse run, ~10 min wall-clock):

AGENT="<agent_id>"; EXP="<experiment_id>"; RID="<experiment_run_id>"

# Phase 0+1
curl -s -X POST -H "Content-Type: application/json" -d "$(cat <<EOF
{"agent_id":"${AGENT}","experiment_id":"${EXP}","run_id":"${RID}",
 "trace_source":{"type":"langfuse"},"storage_config":{"type":"local"}}
EOF
)" http://localhost:18000/api/v1/bucketing-extraction

# Poll
curl -s "http://localhost:18000/api/v1/tasks?experiment_id=${EXP}&experiment_run_id=${RID}"

# Phase 2+3 (consumes every metrics.json under the experiment)
curl -s -X POST -H "Content-Type: application/json" -d "$(cat <<EOF
{"agent_id":"${AGENT}","agent_name":"vaya","experiment_id":"${EXP}",
 "runs_per_fault":5,"storage_config":{"type":"local"}}
EOF
)" http://localhost:18000/api/v1/aggregation-certification

# Poll
curl -s "http://localhost:18000/api/v1/cert-tasks?experiment_id=${EXP}"

---

Certifier Dev Tools

Two helper scripts in scripts/ run the certifier pipeline locally without starting the FastAPI service. Both invoke the same TraceService / BucketPipelineService / CertPipelineService code paths used by the running service, and read LANGFUSE_* / AZURE_OPENAI_* from the root .env.

Dump a Langfuse trace into pipeline-compatible JSON

scripts/dump_langfuse_trace.py fetches a single trace from Langfuse and writes raw_trace.json plus a trace_meta.json summary. This is useful for collecting offline samples that the certifier can later consume via a FileTraceSource.

./scripts/dump_langfuse_trace.py \
  --experiment-id <UUID> \
  --run-id        <UUID>

# or override the dump dir
./scripts/dump_langfuse_trace.py \
  --experiment-id <UUID> --run-id <UUID> \
  --output-dir ./.tmp/sample-001

Run the full certification end-to-end

scripts/run_certification.py runs Phase 0+1+2+3 against one trace and writes outputs into .tmp/ (gitignored) using the same on-disk layout the production API workers create:

.tmp/{agent_id}/{experiment_id}/
  fault-bucketing/{run_id}/
    traces/raw_trace.json
    fault_buckets/*.json
    metrics/*_metrics.json
    ground_truth/*.json
    pipeline_summary.json
  aggregation/aggregation.json
  cert-builder/certification.json   ← final certificate (JSON)
  cert-builder/certification.pdf    ← rendered PDF (auto-generated)

# Look up agent/experiment/run IDs automatically from a Langfuse trace ID
./scripts/run_certification.py --trace-id <LANGFUSE_TRACE_UUID>

# Or pass the IDs directly (no metadata lookup)
./scripts/run_certification.py \
  --agent-id      <UUID> \
  --experiment-id <UUID> \
  --run-id        <UUID> \
  --agent-name    vaya

# Stop after Phase 0+1 (skip aggregation + cert builder)
./scripts/run_certification.py --trace-id <UUID> --skip-cert

Optional flags: --workspace <dir> (default .tmp/), --batch-size (LLM classification batch size, default 10), --runs-per-fault (default 1, used for Phase 2 statistical-significance checks), --no-pdf (skip the PDF render step), and --debug (retain Phase 3 intermediates).

Render a certification report to PDF

The run_certification.py script renders the PDF automatically. To re-render an existing certification.json (or one produced by the running service), use scripts/render_certification_pdf.py:

pip install --user reportlab    # one-time, only needed for PDF output

./scripts/render_certification_pdf.py \
  --input  .tmp/<agent_id>/<exp_id>/cert-builder/certification.json
# → writes certification.pdf next to the input

# or pick a custom output path
./scripts/render_certification_pdf.py \
  --input  certification.json \
  --output ./out/agent-cert.pdf

---

Admin Actions

Build Docker images locally

Build every project-owned image in deploy/images.txt:

make build
# equivalent: ./scripts/build-and-push.sh
# equivalent: ./scripts/setup.sh     ← builds and EXITS; it does not install anything

These defaults build locally without login, push, image pruning, cluster loading, or deployment. The run ends with:

======================================
  All images built successfully!
======================================
  Build-only run: nothing was deployed and no cluster was changed.
  Next: ./scripts/setup.sh --deploy    (configure .env, create the cluster and install ACE)

For a faster rebuild, use make build BUILD_ARGS="--allow-build-cache".

Publish explicitly with ./scripts/build-and-push.sh --push --tag RELEASE-N using a new release tag, or install explicitly with ./scripts/setup.sh --deploy. --kind-load explicitly loads locally built images into KinD. Component build wrappers use the same options and local-only default.

See the script mode guide for every entry point and validation.

---

Community & Contributing

Questions, ideas, or want to help build ACE? Join us on Slack:

👉 AgentCert Slack workspace

See CONTRIBUTING.md for setup, branching, commit, issue, and pull-request guidelines.

---

License

Released under the Apache License 2.0.