Orq.ai Agent Skills
Agent Skills for the full Build → Evaluate → Optimize lifecycle of LLM pipelines on orq.ai.
Skills are multi-step workflows that require reasoning (e.g. build an agent, run an experiment);
Commands are quick actions for immediate results (list traces, show analytics).
Each skill encodes best practices from prompt engineering, agent design, evaluation methodology, and experimentation into repeatable workflows. From creating agents and writing prompts, through trace analysis and dataset generation, to running validated experiments and iterating on results.
Built on the Agent Skills standard format, so it works with any compatible agent (Claude Code, Cursor, Gemini CLI, and others).
Setup
Prerequisites
-
An orq.ai account
-
An API key from Settings → API Keys
export ORQ_API_KEY=your-key-here
Claude Code's orq-skills plugin asks for the key when you install or enable it (change it later with /plugin configure) and keeps it in the system credential store, so you don't need the env var there. Every other harness reads ORQ_API_KEY.
Quick reference
Claude Code plugin
Use this if you want easy access to all components — skills, MCP tools, and trace hooks — in one install. Installed via the orq-ai/assistant-plugins marketplace.
# In Claude Code:
/plugin marketplace add orq-ai/assistant-plugins
# Install all 3 plugins
/plugin install orq-skills@orq-claude-plugin
/plugin install orq-mcp@orq-claude-plugin
/plugin install orq-trace@orq-claude-plugin
| Plugin | What it gives you |
|---|
orq-skills | Skills, commands, and agents for the Build → Evaluate → Optimize lifecycle |
orq-mcp | MCP server registration — Claude can call orq.ai APIs directly |
orq-trace | OTLP tracing hooks that capture Claude Code sessions into orq.ai |
Verify with the interactive onboarding — checks ORQ_API_KEY, MCP reachability, and credentials:
/orq:quickstart
Codex
# Skills (writes to ~/.agents/skills/, which Codex scans by default)
npx skills add orq-ai/assistant-plugins --agent codex -g -y
# orq.ai MCP server (writes [mcp_servers.orq-workspace] to ~/.codex/config.toml)
codex mcp add orq-workspace \
--url https://my.orq.ai/v2/mcp \
--bearer-token-env-var ORQ_API_KEY
Restart Codex. Verify with /mcp (lists orq-workspace) or the prompt "List my orq.ai agents".
Codex plugins don't support slash commands, so /orq:* shortcuts are Claude Code-only — in Codex, describe tasks in natural language to trigger skills.
Skills-only install
Use this when you're on a non-Claude agent (Cursor, Gemini CLI, Cline, Copilot CLI, Codex, Windsurf, and many others), or when you only want the skills without MCP/trace hooks.
npx skills add orq-ai/assistant-plugins
Auto-detects your agent and writes skills to the correct location (e.g. .claude/skills/, .cursor/rules/). Run inside your project directory.
Agent-specific install guides:
Agent Plugins 1.0.0 clients
The repo root is a portable Agent Plugins 1.0.0 plugin: plugin.json manifest, skills in skills/. Point any spec-conformant client at a clone of this repo:
git clone https://github.com/orq-ai/assistant-plugins.git
# then register the cloned directory as a plugin in your client
All skills/* skills are discovered per the spec.
MCP is not yet portable. The bundled mcp.json predates the spec's MCP schema (it declares "type": "http" and carries the API key in a header, neither of which 1.0.0 permits), so a conformant client disables MCP for this plugin and loads skills only. Register the server separately via the MCP-only install. If your client reads the root plugin.json and you previously got MCP through the Claude Code or Codex plugin, expect to lose it until mcp.json is fixed — track that work before relying on this path.
MCP-only install
Use this when you want orq.ai MCP tools in a tool that isn't the Claude Code plugin (Claude Desktop, other MCP-capable clients, or manual Claude Code setup).
# Manual registration in Claude Code
claude mcp add --transport http orq-workspace https://my.orq.ai/v2/mcp \
--header "Authorization: Bearer ${ORQ_API_KEY}"
For other clients, most accept a JSON block with url + headers:
{
"mcpServers": {
"orq-workspace": {
"type": "http",
"url": "https://my.orq.ai/v2/mcp",
"headers": { "Authorization": "Bearer ${ORQ_API_KEY}" }
}
}
}
Manifest validation
tests/scripts/validate-plugin-manifests.sh
Commands
Quick-action slash commands. Use /orq:<command> in Claude Code.
| Command | What It Does | Usage |
|---|
| quickstart | Interactive onboarding — credentials, MCP setup, skills tour | /orq:quickstart |
| workspace | Workspace overview — agents, deployments, prompts, datasets, experiments | /orq:workspace [section] |
| traces | Query and summarize traces with filters | /orq:traces [--deployment name] [--status error] [--last 24h] |
| models | List available AI models by provider | /orq:models [search-term] |
| analytics | Usage analytics — requests, cost, tokens, errors | /orq:analytics [--last 24h] [--group-by model] |
Examples
/orq:workspace agents # Show only agents
/orq:traces --status error --last 1h # Recent errors
/orq:models gpt-4 # Search for GPT-4 variants
/orq:analytics --group-by deployment # Cost per deployment
Skills
Skills are triggered by describing what you need. Claude picks the right skill automatically.
| Skill | What It Does | Documentation |
|---|
| orq-setup-observability | Set up orq.ai observability for LLM applications — AI Router proxy, OpenTelemetry, tracing setup, and trace enrichment | SKILL.md |
| orq-invoke-deployment | Invoke orq.ai deployments, agents, and models via the Python SDK or HTTP API — pass prompt variables, stream responses, and generate integration code | SKILL.md |
| orq-build-agent | Design, create, and configure an orq.ai Agent with tools, instructions, knowledge bases, and memory | SKILL.md |
| orq-build-evaluator | Create validated LLM-as-a-Judge evaluators following evaluation best practices | SKILL.md |
| orq-recommend-evaluators | Recommend the evaluators an agent is missing, from its traces or (with no traffic) its instructions and config, skip ones already attached, and create each only after approval | SKILL.md |
| orq-review-agent | Health-check an agent's setup without changing it, with or without traffic: instructions, tools, skill references, evaluators, model and run health checked against the suite's rubrics, with a suggestion per finding in one HTML report | SKILL.md |
| orq-evaluator-alignment | Align an existing LLM-judge (boolean, categorical, numeric) to human judgment — measure how often it changes its mind, group the least reliable cases and ask a few questions instead of labelling every row, rewrite the judge prompt, and recreate the evaluator after approval |
Workflows
1. Build a New Agent
"I need a customer support agent" → orq-build-agent
"Which evaluators should it have?" → orq-recommend-evaluators
"Create test cases for it" → orq-generate-synthetic-dataset
"Build an evaluator for response accuracy" → orq-build-evaluator
"Run an experiment to get a baseline" → orq-run-experiment
2. Debug Production Issues
/orq:traces --status error --last 24h # Find errors
"Analyze these failures" → orq-analyze-traces
"Fix the prompt based on the failure analysis" → orq-improve-agent
"Re-run the experiment to verify the fix" → orq-run-experiment
3. Improve an Existing Agent
/orq:analytics --group-by deployment # Spot high error rates
"Review the checkout agent's setup" → orq-review-agent
"Analyze traces for the checkout agent" → orq-analyze-traces
"Build evaluators for the failure modes" → orq-build-evaluator
"Generate a dataset covering edge cases" → orq-generate-synthetic-dataset
"Run an experiment and compare" → orq-run-experiment
"Optimize the prompt based on results" → orq-improve-agent
4. Improve an existing Prompt
"My prompt isn't performing well, help me improve it" → orq-improve-agent
"Create test cases to compare before and after" → orq-generate-synthetic-dataset
"Build an evaluator for [specific dimension]" → orq-build-evaluator
"Run an experiment: current vs optimized prompt" → orq-run-experiment
"Refine the prompt based on failure cases" → orq-improve-agent
Data & privacy
- Skills and commands run locally in your agent. They send nothing on their own.
- MCP server (
orq-workspace) talks to https://my.orq.ai/v2/mcp, authenticated with your ORQ_API_KEY as a bearer token. Tool calls you make (agents, deployments, datasets, experiments, traces) go to your orq.ai workspace.
orq-trace (separate plugin, opt-in) sends session, tool-call and LLM-response spans to orq.ai over OTLP, with secrets redacted. See plugins/trace-hooks/README.md to strip content entirely or disable it per session.
Privacy policy: https://orq.ai/legal/privacy · Terms: https://orq.ai/legal/terms-of-service
Links
