Inspect Hawk

---

Start Here

Run evals against an existing Hawk deployment: install the CLI, authenticate, and submit your first eval. See the CLI installation guide and running evaluations guide.

Deploy your own Hawk instance: follow the Quick Start and configuration reference.

Contents

• Install the Hawk CLI
• Deploy Your Own Hawk Instance
• What You Can Do
• What Hawk Deploys
• Architecture
• Documentation
• Repository Structure
• Contributing

Inspect-Hawk is a platform for running Inspect AI evaluations on cloud infrastructure. You define tasks, agents, and models in a YAML config, and Hawk handles everything else: provisioning isolated Kubernetes pods, managing LLM API credentials, streaming logs, storing results in a PostgreSQL warehouse, and serving a web UI to browse them.

Inspect-Hawk is built on Inspect AI, the open-source evaluation framework created by the UK AI Security Institute. Inspect provides the evaluation primitives (tasks, solvers, scorers, sandboxes). Hawk provides the infrastructure to run those evaluations reliably at scale across multiple models and tasks, without manually provisioning machines or managing API keys.

The system is designed for teams that need to run evaluations regularly and at volume. It supports row-level security and access control per model, a managed LLM proxy, and a data warehouse for querying results across runs. It also supports Inspect Scout scans over previous evaluation transcripts — Scout runs automated scanners (e.g. for reward hacking, safety-relevant behavior) across transcripts from completed evaluations, producing structured per-sample scan results.

Demo Video

Install the Hawk CLI

If you already have access to a Hawk deployment, you do not need to read the AWS/Pulumi deployment docs. Install the CLI from PyPI with uv:

uv tool install "hawk[cli]"

Or with pip: pip install "hawk[cli]".

Keep the CLI in sync with your deployment. To match a specific deployment, pin the release (run hawk version to see both sides, or check <API_URL>/version): uv tool install "hawk[cli]==2.0.0". The CLI also checks for drift before each command: by default it warns and asks for confirmation (HAWK_VERSION_CHECK=prompt). Set HAWK_VERSION_CHECK=warn to warn without prompting, block to abort, or off to disable.

Bleeding edge (dev). To install the latest unreleased main instead of a PyPI release: uv tool install "hawk[cli] @ git+https://github.com/METR/hawk#subdirectory=hawk".

Then configure the CLI for your deployment, authenticate, and submit an eval:

export HAWK_API_URL=https://api.hawk.example.com
hawk login
hawk eval-set path/to/your.eval-set.yaml
hawk web

See Running Evaluations, the CLI reference, and hawk/README.md for CLI usage.

Scripts can call the same API without Click via hawk[client].

If you do not already have an eval-set YAML, follow the running evaluations guide to create one.

Deploy Your Own Hawk Instance

If you do not have a Hawk deployment to connect to, the Quick Start walks through deploying Hawk on AWS from zero. You will need an AWS account, a domain name, Pulumi, Docker, Python, Node.js, and uv.

Use the configuration reference for DNS, authentication, networking, and optional integrations.

What You Can Do

• 📋 One YAML, full grid. Define tasks, agents, and models. Hawk runs every combination.
• ☸️ Kubernetes-native. Each eval gets its own pod and fresh virtualenv. Sandboxes run in separate pods with network isolation.
• 🔑 Built-in LLM proxy. Managed proxy for OpenAI, Anthropic, and Google Vertex with automatic token refresh. Bring your own keys if you prefer.
• 📡 Live monitoring. hawk logs -f streams logs in real-time. hawk status returns a structured JSON report.
• 🖥️ Web UI. Browse eval sets, filter samples by score and full-text search, compare across runs, export to CSV.
• 🔍 Scout scanning. Run scanners over transcripts from previous evals.
• 🗄️ Data warehouse. Results land in PostgreSQL with trigram search and covering indexes.
• 🔒 Access control. Model group permissions gate who can run models, view logs, and scan eval sets.
• 💻 Local mode. hawk local eval-set runs the same config on your machine for debugging.

What Hawk Deploys

When you run pulumi up, Hawk creates the following infrastructure on AWS:

Component	Service	Purpose
Compute (evals)	EKS	Runs evaluation jobs as isolated Kubernetes pods
Compute (API)	ECS Fargate	Hosts the Hawk API server and LLM proxy
Database	Aurora PostgreSQL Serverless v2	Results warehouse with IAM auth, auto-pauses when idle
Storage	S3	Eval logs, written directly by Inspect AI
Event processing	EventBridge + Lambda + AWS Batch	Tags logs and imports them into the warehouse
Web viewer	ECS Fargate	Browse and analyze evaluation results (static SPA)
Networking	VPC + ALB	Internet-facing load balancer with TLS (configurable)
DNS	Route53	Service discovery and public DNS

Aurora can auto-pause and Karpenter can scale workload nodes to zero when idle. The EKS control plane, Karpenter controller node group, ECS services, ALB, and networking remain provisioned.

Architecture

flowchart TD
    CLI["<b>hawk CLI</b>"]
    Client["<b>hawk.client</b><br/><i>scripts · hawk[client]</i>"]
    API["Hawk API Server<br/><i>FastAPI on ECS Fargate</i>"]
    EKS["Kubernetes (EKS)<br/><i>Scaled by Karpenter</i>"]
    Runner["Runner Pod<br/><i>Creates virtualenv, runs inspect_ai.eval_set()</i>"]
    Sandbox["Sandbox Pod(s)<br/><i>Isolated execution · Cilium network policies</i>"]
    S3[("S3<br/><i>Eval logs</i>")]
    S3Events["EventBridge<br/><i>S3 Object Created</i>"]
    JobStatusUpdated["job_status_updated<br/><i>Lambda · tags files, emits events</i>"]
    EvalEvents["EventBridge<br/><i>EvalCompleted</i>"]
    Importer["eval_log_importer<br/><i>AWS Batch · imports to warehouse</i>"]
    DB[("Aurora PostgreSQL<br/><i>Results warehouse</i>")]
    Viewer["Web Viewer<br/><i>ECS Fargate · Browse, filter, export</i>"]
    Middleman["Middleman<br/><i>LLM Proxy</i>"]
    LLMs["LLM Providers<br/><i>OpenAI · Anthropic · Google · etc.</i>"]

    CLI --> API
    Client --> API
    API -- "Validates config & auth<br/>Creates Helm release" --> EKS
    EKS --> Runner
    EKS --> Sandbox
    Runner -- "Writes logs" --> S3
    Runner <-- "API calls" --> Middleman
    Middleman --> LLMs
    S3 --> S3Events
    S3Events --> JobStatusUpdated
    JobStatusUpdated --> EvalEvents
    EvalEvents --> Importer
    Importer --> DB
    Viewer -- "Browser calls" --> API

See the Architecture docs for the full component breakdown.

Documentation

Full documentation lives at hawk.metr.org:

• Quick Start — deploy Hawk on AWS from zero
• Installation — install the CLI and run your first eval
• Running Evaluations — eval set configs, secrets, local runs
• Running Scans — Inspect Scout scans over transcripts
• CLI Reference — full command reference
• Python client — hawk[client] API
• Configuration Reference — every Pulumi config key
• Infrastructure — architecture, security, database, deployment, teardown

Repository Structure

infra/        Pulumi infrastructure (Python) — VPC, EKS, ALB, ECS, RDS, Lambdas
hawk/         Hawk application (Python + React) — CLI, hawk.client, API, runner, core, web viewer
middleman/    LLM proxy (OpenAI, Anthropic, Google Vertex)
docs/         Documentation site (published to hawk.metr.org)

Contributing

See the Contributing guide for developer setup, local development (Docker Compose or Minikube), testing, and code quality guidelines.

License

MIT