CortexOps
Reliability infrastructure for AI agents.
Evaluate · Observe · Operate — for LangGraph, CrewAI, and AutoGen.
Architecture
Supported Frameworks
Tests
The problem
You deployed an agent. You have no idea if it regressed overnight.
No standard eval format. No failure traces. No CI gate before the next prompt change ships.
CortexOps fixes that.
Quickstart
pip install cortexops
from cortexops import CortexTracer, EvalSuite
# Wrap your LangGraph app — zero refactor required
tracer = CortexTracer(project="payments-agent")
graph = tracer.wrap(your_langgraph_app)
# Run evaluations against a golden dataset
results = EvalSuite.run(
dataset="golden_v1.yaml",
agent=graph,
)
print(results.summary())
# CortexOps eval — payments-agent
# Cases : 9 (7 passed, 2 failed)
# Task completion : 91.4%
# Tool accuracy : 97.0/100
# Latency p50/p95 : 42ms / 187ms
# Failed cases:
# - escalation_router: tool_call_mismatch (score 41)
Hosted API (Pro)
The free tier runs evals locally. The Pro tier adds hosted trace storage,
a live dashboard, Slack alerts, and LLM-as-judge scoring.
Get an API key at getcortexops.com
Then point the SDK at the hosted API:
from cortexops import CortexTracer, EvalSuite
tracer = CortexTracer(
project="my-agent",
api_key="cxo-your-key-here",
api_url="https://api.getcortexops.com"
)
graph = tracer.wrap(your_langgraph_app)
Traces are stored for 90 days. View them in the
dashboard or via the
API docs.
When a production trace fails, promote it to a golden dataset case in one click —
then run the same case in CI before the next deploy.
Pricing
| Tier | Price | What you get |
|---|
| Free | $0 | Local evals, CLI, GitHub Actions gate |
| Pro | $49/seat/mo | Hosted traces, Slack alerts, dashboard, LLM judge |
| Enterprise | Custom | VPC, SSO, SLA |
Start a Pro trial: getcortexops.com/#pricing
Golden dataset format
Define test cases in YAML. Run them locally or in CI.
# golden_v1.yaml
version: 1
project: payments-agent
cases:
- id: refund_lookup_01
input: "What is the status of refund REF-8821?"
expected_tool_calls: [lookup_refund]
expected_output_contains: ["approved", "REF-8821"]
max_latency_ms: 3000
- id: dispute_escalation_01
input: "I was charged twice — this is unauthorized"
expected_tool_calls: [classify_dispute, route_escalation]
expected_output_contains: ["escalated"]
max_latency_ms: 5000
CI eval gate
Add the CortexOps Eval Gate GitHub Action to a workflow:
jobs:
eval:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write # for the sticky PR comment
steps:
- uses: actions/checkout@v4
- uses: ashishodu2023/cortexops-eval-action@v1
with:
dataset: golden_v1.yaml
agent: agent:build_agent()
fail-on: |
task_completion < 0.90
tool_accuracy < 0.95
It writes a report (metrics, thresholds, failed cases) to the job summary, keeps one comment on the PR up to date, and fails the job if any condition is true. Without the action, the same gate is one CLI call:
cortexops eval run --dataset golden_v1.yaml --agent "agent:build_agent()" \
--fail-on "task_completion < 0.90" --output-json report.json --summary-md summary.md
GitLab users can include ci-templates/gitlab/cortexops-eval.gitlab-ci.yml; a cortexops-eval pre-commit hook is also available.
Or gate from pytest (pip install "cortexops[pytest]"):
import cortexops
@cortexops.eval_gate("golden_v1.yaml", fail_on="task_completion < 0.90")
def test_payments_agent():
return build_agent()
Repo structure
cortexops/
├── sdk/ # pip install cortexops
│ ├── cortexops/
│ │ ├── tracer.py # CortexTracer — wraps LangGraph / CrewAI
│ │ ├── eval.py # EvalSuite — golden dataset runner
│ │ ├── metrics.py # task_completion, tool_accuracy, latency, hallucination
│ │ ├── models.py # Pydantic data models
│ │ └── client.py # HTTP client for hosted API
│ └── tests/
├── backend/ # FastAPI + Celery + SQLite/Postgres
│ ├── app/
│ │ ├── main.py
│ │ ├── routers/ # /v1/evals, /v1/traces
│ │ ├── models/ # DB records + API schemas
│ │ └── worker/ # Celery async eval tasks
│ └── Dockerfile
├── frontend-new/ # React dashboard (app.getcortexops.com)
├── docs-site/ # Static docs (docs.getcortexops.com)
├── examples/
│ └── langgraph_payments/ # Full runnable demo
│ ├── agent.py
│ ├── golden_v1.yaml
│ └── run_eval.py
└── docker-compose.yml
Run the full stack locally
git clone https://github.com/ashishodu2023/cortexops
cd cortexops
# Start API + worker + Redis
docker compose up --build
# In another terminal — run the demo eval
cd examples/langgraph_payments
pip install -e ../../sdk/
python run_eval.py
# API docs at http://localhost:8000/docs
# Dashboard at http://localhost:3000
Supported frameworks
| Framework | Status |
|---|
| LangGraph | Stable |
| CrewAI | Stable |
| AutoGen | Beta |
| LlamaIndex agents | Coming soon |
| Custom callables | Supported via CortexTracer.wrap() |
Built-in metrics
| Metric | What it checks |
|---|
task_completion | Agent produced a valid, non-error output |
tool_accuracy | Expected tool calls were actually made |
latency | Response within max_latency_ms budget |
hallucination | Detects fabrication signals in output |
Add custom metrics by subclassing cortexops.Metric.
Roadmap
v0.5.0 — shipped
v0.5.x — shipped in 0.5.1
v0.6.0 — distribution and CI-first
v0.7.0 — ecosystem
v1.0.0 — enterprise
Want to work on any of these? See open issues or open a discussion.
Contributing
git clone https://github.com/ashishodu2023/cortexops
cd cortexops/sdk
pip install -e ".[dev]"
pytest tests/ -v
See CONTRIBUTING.md. Issues labeled good first issue are a great place to start.
To report a vulnerability, see SECURITY.md.
Citation
@software{cortexops2025,
author = {Ashish, et al.},
title = {CortexOps: Reliability Infrastructure for AI Agents},
year = {2025},
url = {https://github.com/ashishodu2023/cortexops},
}
License
MIT — see LICENSE.
