Coval MCP Server

The official Model Context Protocol server for Coval - the AI evaluation platform.

This MCP server allows AI assistants like Claude, ChatGPT, Codex, and Cursor to interact with Coval's evaluation APIs, enabling you to:

• Launch and monitor evaluation runs
• Manage AI agents and test sets
• Retrieve evaluation metrics and results
• Inspect saved reports and recurring evaluation schedules

Hosted connector

The hosted connector is the recommended way to use Coval from Claude, ChatGPT, and other remote MCP clients. It uses OAuth, so users sign in to Coval and select the organization the connector can access. No Coval API key is copied into the client.

Standard MCP endpoint: https://mcp.coval.dev/mcp

Claude hosted endpoint: https://mcp.coval.dev/claude/mcp

Claude

1. Open Customize > Connectors. Team and Enterprise owners add the connector first from Organization settings > Connectors.
2. Add a custom web connector using https://mcp.coval.dev/claude/mcp.
3. Select Connect, sign in to Coval, and choose the Coval organization to authorize.
4. Enable the connector in a conversation and ask Claude to use Coval or Sofia.

See Claude's remote MCP connector guide for current plan and administration requirements.

ChatGPT

1. Enable developer mode for your ChatGPT workspace account.
2. From Settings > Apps, create an app using https://mcp.coval.dev/mcp and OAuth.
3. Select Scan Tools, sign in to Coval, and choose the Coval organization to authorize.
4. Create the app, then enable it in a conversation.

See ChatGPT's MCP app guide for current plan and workspace requirements.

Codex

Use Codex 0.144.2 or newer. Current Codex Desktop releases include a compatible client. For the CLI, update before connecting:

npm install -g @openai/codex@latest
codex mcp add coval --url https://mcp.coval.dev/mcp

When the browser opens, sign in to Coval, select the organization Codex should access, and select Allow. Codex Desktop, the Codex CLI, and the IDE extension share MCP configuration on the same machine, so the connection is available across those clients after they restart.

In Codex Desktop, you can also add Coval from Settings > MCP servers as a Streamable HTTP server using https://mcp.coval.dev/mcp, then select Authenticate.

If OAuth returns invalid_scope, first confirm codex --version is 0.144.2 or newer. Older clients registered a dynamic OAuth client without all the scopes they requested during consent. After updating, clear the stale registration and reconnect:

codex mcp logout coval
codex mcp remove coval
codex mcp add coval --url https://mcp.coval.dev/mcp

No API key, manual scope list, or oauth_resource override is required. See Codex's MCP documentation for current client setup details.

Local installation

npx @coval/mcp-server

Use the local stdio server for service accounts, automation, and clients that do not support remote OAuth.

Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "coval": {
      "command": "npx",
      "args": ["-y", "@coval/mcp-server"],
      "env": {
        "COVAL_API_KEY": "your_api_key_here"
      }
    }
  }
}

Cursor

Add to .cursor/mcp.json in your project:

{
  "mcpServers": {
    "coval": {
      "command": "npx",
      "args": ["-y", "@coval/mcp-server"],
      "env": {
        "COVAL_API_KEY": "your_api_key_here"
      }
    }
  }
}

Get your API key from app.coval.dev/settings

Available Tools

Tool	Description
list_agents	List all agents in your workspace
get_agent	Get details of a specific agent
create_agent	Create an agent configuration
update_agent	Update an agent configuration
list_runs	List evaluation runs
get_run	Get details of a specific run
create_run	Start a new evaluation run
list_test_sets	List available test sets
get_test_set	Get test set details
create_test_set	Create a test set
list_test_cases	List test cases in a test set
get_test_case	Get test case details
create_test_case	Add a test case to a test set
update_test_case	Update a test case
list_metrics	List evaluation metrics
get_metric	Get metric details
create_python_metric_draft	Create an immutable Python metric draft
get_python_metric_draft	Get a Python metric draft and fingerprint
create_python_metric_preview	Start an asynchronous Python metric preview
get_python_metric_preview	Poll a Python metric preview for its result
cancel_python_metric_preview	Cancel an in-progress Python metric preview
publish_python_metric_draft	Publish a draft with successful preview IDs
list_personas	List available personas
get_persona	Get persona details
list_reports

Example Usage

Once connected, you can ask Claude things like:

"Show me my recent evaluation runs" "List all my agents" "Run an evaluation of my customer-support-agent against the billing-inquiries test set" "What are the metrics for run abc123?" "Create an immutable Python metric draft, preview it on a conversation, and publish it after the preview succeeds." "Show the first 20 rows of my regression report for the task-success metric" "Create a private report comparing these completed runs by agent" "Show my enabled evaluation schedules and the latest runs from the nightly regression" "Create a disabled weekday schedule from my regression run template" "Use consult_sofia to analyze my latest failed run and recommend the most useful next test."

The same tools, including consult_sofia, are available through every supported endpoint and transport:

• Standard remote Streamable HTTP: https://mcp.coval.dev/mcp using OAuth. This is the recommended connection for ChatGPT, Codex, and other hosted MCP clients.
• Claude remote Streamable HTTP: https://mcp.coval.dev/claude/mcp using the same OAuth flow and tool surface, with Claude-specific risk annotations.
• Local stdio: npx @coval/mcp-server with COVAL_API_KEY, for service accounts and local development.

The hosted connector can access only the Coval organization selected during OAuth consent. Remove the connector from the client or revoke its Coval access when it is no longer needed.

Development

# Install dependencies
npm install

# Build
npm run build

# Test locally with MCP Inspector
npm run inspector

# Run tests
npm test

# Check the public health and OAuth discovery endpoints
npm run check:remote

Complete transcript exports

For a complete run transcript request, consult_sofia can return a native MCP resource_link after collecting every call. Read that URI with resources/read. Clients without resource-link support can use read_transcript_export: concatenate its text fragments exactly, carrying next_offset and version as expected_version until complete=true, then JSON-decode the document. Offsets count Unicode code points; preserve whitespace while concatenating. Each fragment contains complete code points and can be appended as UTF-8 bytes. The document includes a final completion marker, per-call unavailable reasons, source order, original transcript fields, timing and tool ownership.

Exports use the currently authenticated API client's Default workspace. Every read checks current authorization; the URI contains identifiers and fingerprints, not credentials or a public download token. The export is reconstructed from current API reads, not stored as a historical snapshot. Source changes are reported explicitly, and continuation refuses to mix versions. A new authenticated server can read the same URI if it still has access to the run.

Collection supports up to 100 calls, 2 MiB of serialized content and a 30-second deadline. Exceeding a limit fails explicitly instead of returning a sample. A missing transcript is listed with a reason. Authentication, incomplete inventory and mismatched run membership fail the export. Readiness comes from the collector; Sofia's export request or prose alone does not prove delivery. Unsupported older Sofia servers continue returning their existing consultation response.

API Coverage Audit

scripts/audit_api_coverage.py compares the live public OpenAPI catalog (https://api.coval.dev/v1/openapi) with what the MCP server actually calls. It has two layers:

• Operations: every published operation is either reached by a registered tool (traced from src/tools/ through src/client.ts, including private client helpers) or recorded in api-coverage.toml with a reason.
• Request fields: for every covered POST/PATCH/PUT operation, every published application/json request-body property is either declared on the body type the client sends or recorded with a reason. A field the client sends but the API does not publish is also reported, because the API may ignore it.

The MCP server exposes a curated tool set, so most operations are reviewed gaps. The audit's job is to surface what the API adds or changes so each change gets a decision.

python3 -m pip install --requirement scripts/requirements-audit.txt
python3 -m unittest discover --start-directory scripts --pattern 'test_*.py'
python3 scripts/audit_api_coverage.py --write-markdown api-coverage-report.md

The audit fails for new or stale gaps, a stale [snapshot] in the manifest, or tool calls to client methods that do not exist. Resolve a failure by exposing the operation or field through a tool, or by recording it in api-coverage.toml with a reason, then regenerate the report.

Every Monday at 10:00 UTC, .github/workflows/api-parity-audit.yml refreshes the report and opens or updates one rolling pull request from chore/weekly-api-parity. This is the same mechanism and schedule the Coval CLI uses. The live audit is strict on that pull request and advisory on others, so an API change never blocks unrelated work. Push reconciliation commits onto the rolling pull request. While that pull request is open, the next weekly run leaves a branch with commits not made by the bot unchanged and comments on the pull request instead of resetting it, so merge or rebase that pull request to let later runs pick up new API changes. Once the pull request is merged or closed, the next run rebuilds the branch from main. If the automation itself fails, the run opens an issue.

Container runtime

The hosted container targets Linux x86-64. Build it with docker build --platform linux/amd64 ., including on Apple Silicon. It uses the checksum-pinned official Node.js 26.11.1 musl distribution with OpenSSL 3.5.9 and Undici 8.11.2; other container architectures fail explicitly. The local npm package still supports the Node versions declared in package.json.

Environment Variables

Variable	Required	Default	Description
COVAL_API_KEY	Stdio	-	Coval API key for the local stdio transport
COVAL_API_BASE_URL	No	https://api.coval.dev/v1	API base URL
SOFIA_DELEGATION_ORIGIN	No	Derived from COVAL_API_BASE_URL	Overrides the expected Sofia origin used to validate delegation URLs
LOG_LEVEL	No	info	Logging level
MCP_ALLOWED_ORIGINS	No	Claude and OpenAI web origins	Comma-separated exact browser origins allowed to call /mcp or /claude/mcp; clients that omit Origin remain supported
OPENAI_APPS_CHALLENGE	No	-	OpenAI plugin-portal domain verification token served as plain text from /.well-known/openai-apps-challenge
DD_ENV	No	local	Deployment environment included in structured hosted-request logs
DD_SERVICE	No	coval-mcp-server	Service name included in structured hosted-request logs
DD_VERSION	No	COVAL_MCP_SOURCE_SHA or unknown	Immutable release identifier included in structured hosted-request logs

Documentation

• Coval Documentation
• MCP Protocol
• API Reference
• Privacy Policy
• Connector directory submission checklist

License

MIT

Support

• GitHub Issues
• Coval Support

Package releases

A reviewed PR that increases the stable version in package.json and package-lock.json triggers .github/workflows/publish.yml after CI passes on that exact main commit. The workflow builds and tests the package, publishes the verified archive with npm provenance, checks registry integrity, and creates a matching GitHub release. Commits without a version increase do not publish. Re-running the successful main CI run retries a failed release; an existing npm version is accepted only when its archive integrity matches. Hosted MCP deployment follows its separate release process.

Before the first automated package release, a package maintainer must configure npm trusted publishing for @coval/mcp-server: GitHub owner coval-ai, repository mcp-server, workflow filename publish.yml, no environment, and permission to publish. This uses GitHub OIDC with Node 24 and requires no npm token secret. Until that trust is configured, version-bump releases fail closed at npm authentication.